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ABSTRACT 


The objectives of this thesis were to validate and 
evaluate the Baroudi and Orlikowski [2] short-form UIS 
instrument as a surrogate measurement of effectiveness of the 
Department of Defense's Composite Health Care System (CHCS). 
Hurd [1], using the short-form UIS instrument at the Naval 
Hospital Charleston, suggested that a fundamental change in 
the instrument's structure due to significant impact of 
contractor's services with the CHCS. The three Navy CHCS 
operational testing and evaluation hospitals were used in this 
study. 


The short-form UIS instrument was found to be inadequate 
for evaluation of overall UIS of the CHCS. Limitations found 
were: a fundamental change in the factor structure; unevenly 
distributed variables in a couple of factors; low reliability 
in one factor; no assessment of user interface with the 
system; and inconsistent convergent validity findings. A 
proposed re-design is presented. 


Statistically significant differences in UIS' between 
groups and sites were found for the four factor solution. 
Participants were divided into three functional user groups: 
Physicians, Medical Support, and Administrative Support. All 
three groups were satisfied with the local Management 
Information Department staff and services and the system's 
OuEpuics All three groups were dissatisfied with the 
contractor services. Physicians were significantly less 
satisfied in three out of the four factors. No significant 
correlation between time of use of the system and UIS was 
found, except for the Administrative group's positive 
correlation with the system's output. 
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I. INTRODUCTION 


Modern health care 1S becoming ever increasingly complex 
and costly in its delivery. The Department of Defense in its 
effort to combat these issues has put considerable effort and 
time into health care computer-based information systems. 
The latest system development underway is the Composite 
Health Care System (CHCS). This system integrates various 
health care components within the hospital-based health care 
catchment area. 

Good management practices deem assessment of effectiveness 
of any implementation effort. Effectiveness of a computer- 
based information system can be defined as meeting the needs 
of the users it was intended to help. A commonly used 
assessment of system effectiveness used in the literature is 
the surrogate measure of user information satisfaction (UIS)-- 
an empirical psychometric approach. Hurd [1] conducted an 
evaluation of UIS at the Naval Hospital Charleston, South 
Carolina--one of the Navy's three naval hospital operational 
Peeting and evaluation (OT&E) sites for CHCS. In his 
research, he used the UIS instrument and evaluation 
methodology developed and validated by Baroudi and Orlikowski 
[2]. The Baroudi and Orlikowski UIS instrument explained 
overall satisfaction to be the summation of each item 


(question) score which is the composition of three factors. 


According to Baroudi and Orlikowski [2], the three factors 
that make up overall UIS are: 1) Electronic Data Processing 
(Management Information Department in naval hospitals) Staff 
and Services, 2) Information Product Output, and 3) User 
Knowledge and Involvement of the system. Hurd [1] separated 
the one factor dealing with Management Information Department 
staff and services into essentially two factors due to the 
fact that two of the questions dealt with services not 
provided internally--rather they are performed by an external 
vendor. This resulted ina four factor instrument, one that 
demonstrated the additional impact of contractor services on 
overall UIS in this setting. This fundamental change in the 
instrument's structure would tend to suggest invalidation of 
the overall UIS score, especially since the original 
instrument was specifically designed (content of the 


questions) to support the three factors mentioned above [2]. 


A. RESEARCH OBJECTIVES 

The primary objective of this research is to evaluate and 
validate the Baroudi and Orlikowski UIS instrument (to be 
referred to as the short-form UIS instrument throughout this 
paper) uSing a particular setting and system--the CHCS at 
naval hospitals. Hurd's data will be included with the data 
collection samplings from the two other naval hospitals 
designated as CHCS OT&E sites. Questions to be answered are: 
1) does the four factor (Management Information Department 
Staff and Services, Contractor Services, Information Product 
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Output, and User Knowledge and Involvement) exist using 
common factor-analytic techniques?; 2) is there equivalence 
(or invariance) of the four factor instrument across’~ the 
three Navy CHCS OT&E sites?; and 3) using follow-up random 
interviews of system users, literature review, and empirical 
judgment, does the short-form UIS instrument adequately assess 
overall satisfaction? 

Data collected from this research can be of additional 
informational value. Hurd's [1] findings indicated that 1) 
physicians were least satisfied with training; 2) physicians 
were least satisfied with understanding the system; 3) that 
all his designated user groups (physicians, ancillary 
Support, and administration) were dissatisfied with the 
services provided by the contractor (i.e., handling and speed 
of implementation of system changes); and 4) there was no 
correlation between the length of time of system use and 
overall UIS score for all user groups. 

Secondary objectives will be to evaluate the foilowing: 1) 
do physicians significantly stand out as less satisfied as 
compared with other user groups?; 2) within user groups are 
there significant differences between location sites?; and 3) 
is there a correlation between length of time of use of the 


CHCS system and UIS? 


B. OVERVIEW 
Chapter II, discusses a brief review of the alternative 
measures of system effectiveness, some of the problems 
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associated with purely quantitative measures (i.e., economic 
analyses), and the development of the attitude psychometric 
instrument that is used in this. study. Chapter III, 
discusses the research methodology used for the primary and 
secondary objectives, including a summary of the CHCS and the 
setting for this study. Chapter IV, presents the findings 
from the data collection, tabulation, and testing techniques 
employed. Chapter V, discusses the significant results and 
their impact towards the primary and secondary objectives. 
Lastly, Chapter VI, presents the conclusions derived from 


this study, as well as those issues for future research. 


II. USER INFORMATION SATISFACTION AND ITS MEASUREMENT 


A. INTRODUCTION 

Computer-based information systems (CBIS) can be very 
expensive. As such, administrators and researchers have long 
attempted to identify a meaningful means to evaluate CBIS 
effectiveness. The most common approach suggested by 
Information Systems (IS) literature for measuring CBIS 
effectiveness has been user information satisfaction (UIS). 
Ives et al., define UIS as "the extent to which users believe 


the IS available to them meets their information requirements 


(3, p.785]." UIS is a perceptual or subjective "surrogate" 
measure of system effectiveness. Objective, quantitative 
measures, uSing economic’ analysis (1.e., cost-benefit 


determinations) have been reported to present problems in 


quantifying the actual benefits of a CBIS. 


B. METHODS TO ASSESS EFFECTIVENESS 

Generally speaking, measurement of CBIS effectiveness 
falls into two categories: economic and psychometric. 
Ideally, the CBIS's effectiveness or value to the 
organization would be evaluated using an objective economic 
analysis. The costs associated with the system operation 
would be subtracted from the actual benefits from using the 
system equated in monetary terms resulting in the net 
monetary valuation of effectiveness to the SEGanizacion. 
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However, Ives et al., and many other researchers have argued 


that this is not a easy task due to: 1) diftficultyeee 


recognize and quantify intangible benefits; 2) certain 
benefits such as decision support ad hoc queries are 
Virtually impossible to quantify; and 3) even. when 


organization's attempt to quantify these items, they are 
usually highly subjective, and undocumented in how derived,or 
unavailable for research [1-10]. Approaches that have been 
advocated for measuring and improving CBIS effectiveness 
include usage estimation [11], incremental performance in 
decision making effectiveness [12], cost-benefit analysis 
(13], information economics [14], utility analysis [15], the 
analytic hierarchy approach [16], information attribute 
examination [17], and user information satisfaction. 

A number of researchers have studied UIS as the measure of 
CBIS effectiveness [1-10] and [18-25]. Powers and Dickson 
[7}, ena CerniiownT >) Suggest that user satisfaction (a 
psychometric measure) is the most important element in 
determining CBIS success. According to Conrath and Mignen 
[20], the key contribution in developing a tool for measuring 
and analyzing computer user satisfaction has been the 


research work of Bailey and Pearson. 


C. DEVELOPMENT OF THE UIS QUESTIONNAIRE INSTRUMENT 

Bailey and Pearson's [18] research involved a literature 
review of 22 studies of the computer/interface to establish an 
initial list of variables that might affect user satisfaction. 
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From the literature review, they identified 36 variables. 
Next, they had three data processing professionals review the 
list, wherein the list was expanded to 38 variables. The 38 
variable list was empirically compared to interview responses 
from 32 managers of information systems in eight different 
organizations. After this interview process, one additional 
variable was added to the list for a total of 39 variables. 
Using the bidirectional model of Wanous and Lawler [26] which 


suggests that satisfaction is the sum of one's positive and 


negative reactions to a set of variables, and using 
semantic differential technique [27], Bailey and Pearson 
added dimensionality and intensity of an individual's 


reaction to each variable based on the use of adjectives to 
explain the user's perceptions. A seven interval Likert-type 
measurement scaling model was used to enumerate the adverbial 
qualifiers: extremely, quite, slightly, neither/equally, 
slightly, quite, and extremely for each of the negative to 
positive (bipolar) adjective pair. Each variable had four 
adjective pairs, one pair to rate satisfactory versus 
unsatisfactory, and one pair to rate whether the variable was 
important versus unimportant. The previously interviewed 32 
managers were asked to fill out the questionnaire, with 29 
being completed and returned. Although the sample size was 
small, Bailey and Pearson's instrument presented significant 
progress toward the development of a standard measure of 


UIS. [18] 


Bailey and Pearson's [18] research instrument or construct 
for measuring UIS using their 39 variable questionnaire, set 
the stage for further research. They recommended further 
research and validation efforts in a wide variety of user 
environments, and the application of factor analysis to see 
if and when the list of variables can be reduced. 

Ives, Olson, and Baroudi [3], chose to undertake a 
replication of Bailey and Pearson's findings to reinforce the 
validity of the instrument through further tests, and reduce 
the length of the instrument in an attempt to produce a 
standard "short-form" instrument. Their research was based 
on the completed questionnaires from 200 production managers 
in U.S. manufacturing organizations selected from a 
commercially obtained mailing list. From their research, 
the list of variables from the original Bailey and Pearson 
instrument were reduced to 22, and the four bipolar adjective 
pairs reduced to two pairs. Factor analysis testing was 
conducted on the 22 variable instrument to see if the 
information could be reduced into fewer latent variables 
(unobserved variables) or factors. Essentially, four factors 
were identified: Electronic Data Processing (EDP) Staff and 
Services, Information Product Output, User Knowledge and 
Involvement, and Vendor Support. Vendor Support was dropped 
due to only one variable loading into this factor, and the 
researcher's belief at the time that it was not a significant 


component to overall user information satisfaction 


assessment. Their reliability, validity, and correlation data 
provided substantial evidence that the shortened form 
questionnaire iS a sound general measure of Bailey and 
Pearson's original UIS concept. 

Ives et al. [3], recommended further research to validate, 
extend, and disseminate the instrument. Additionally, they 
recommended a change in the instrument format. The original 
Bailey and Pearson [18] instrument, and the Ives, Olson, and 
Baroudi [3] instrument had each of the bipolar adjective pairs 
in the same directional format, scored negatively to the 
right, and scored positively to the left. Reliability would 
be further increased by mixing up the direction of some of 
the adjective pairs for some of the variables [3]. 

Baroudi and Orlikowski [2] further refined the short form 
of the UIS instrument previously researched by Ives et al. [3] 
into a 13 question (variable) measure of UIS. The 13 
variables were selected because they believed the selected 
variables displayed the most desirable psychometric properties 
for the three UIS factors (EDP Staff and Services, Information 
Product Output, and User Knowledge and Involvement). One of 
the objectives of their study was to develop a standard 
"short-form" UIS instrument. As with the previous 
instruments, each bipolar adjective pair is scored uSing a 
seven interval Likert-type measurement scaling model with 
values ranging from -3 (extremely dissatisfied) to +3 


(extremely satisfied), and zero indicating a neutral 


response. Each variable (question) is scored by taking the 
average of the two bipolar adjective pair scores. They mixed 
up the direction of a number of the adjective pairs as 
recommended from previous research. 

Baroudi and Orlikowski [2] specifically picked the 
variables in the short-form (Appendix A, Part B) that 
measured the three factors that Ives, Olson, and Baroudi [3] 
found to comprise overall UIS. Three subtotals representing 
the three factors were calculated by averaging the variables 
(questions) that comprised each (ranging from -3 to +3). 
Questions 1, 2, 6, 11, and 12 factor loaded in the assessment 
of the attitude and responsiveness of the EDP staff as well as 
the quality of the relationship between the user and the EDP 
staff for scoring the first factor. Questions 7, 8, 9, 10, 
and 13 factor loaded in the assessment of the quality of 
output delivered by the information system for the scoring of 
the second factor. Questions 3, 4, and 5 factor loaded in 
the assessment of the quality of training provided, 
understanding of the system, and participation in the system's 
development for scoring the third factor. The overall UIS 
score was determined by the summation of each of the 13 
variable (question) scores. The total score could range from 
—39 Cems 7. 

Data was collected from 368 employees, mostly clerical and 
Support personnel, of 26 New York area organizations using 


transaction processing computer systems. Follow-up interview 
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measure of UIS from five of the organizations was conducted in 
fdigmerem to internal construct validity and reliability 
analysis. The data indicated that the interview assessments 
of user satisfaction or dissatisfaction correlated well with 
the satisfaction scores obtained by the short-form instrument 
and provides some evidence for the instrument's convergent 
validity. 

Igbaria and Nachman [5] provided further validation of the 
Baroudi and Orlikowski short-form instrument in their study 
using a different sample of users. They performed a second 
order factor analysis to test for underlying homogeneity in 
the first order three factors that were extracted to assess 
overall UIS. Their results confirmed the homogeneity of the 
items and the appropriateness of combining the 13 questions 
as an overall measure of UIS. Additionally, they added a 
demographic section to the short-form instrument to assess 
the role that individual characteristics had with UIS. Based 
on previous research, they included computer experience, 
education, age, gender, organizational level, and time of use 
of computer system as user variables for UIS. From their 
research, they found that education, organizational level and 
gender were not correlated with UIS. However, their findings 
indicated a significant positive relationship between UIS and 
length of time using the CBIS. 

Hurd [1] used the Baroudi and Orlikowski [2] short-form 


UIS instrument in his study of a specific CBIS (the Composite 
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Health Care System) at a particular site, the Naval Hospital 
Charleston, South Carolina. Hurd also used a demographic 
section with essentially the same salient user variables as 
Igbaria and Nachman [5], modified to meet the site under 
study (Appendix A, Part A). Hurd's findings suggested no 
correlation of overall UIS with time of system use. However, 
looking at the three groups of users (phySicians, ancillary 
personnel, and administrators), Hurd noted that there were 
identifiable trends. Physician's overall UIS tended to 
remain essentially unchanged with time of system use. 
Administrators exhibited a positive trend, whereas the 
ancillary personnel exhibited a negative trend between UIS and 
time of system use. 

The instrument used by Hurd (Appendix A) was utilized in 
this study to extend the study of CHCS user satisfaction at 
the two other CHCS OT&E naval hospitals for validation of the 


instrument, comparison, and time of use correlation testing. 
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III. RESEARCH METHODOLOGY 


A. INTRODUCTION 

This study has two main objectives. The primary objective 
is to evaluate the construct validity of the short-form UIS 
instrument. The second objective 1s a follow on from the 
primary objective. The findings from the currently used 
short-form UIS instrument will be presented and evaluated in 
light of the construct validity for user group/site UIS 
comparisons. In addition, the correlation between the length 
of time the user has used the CHCS and UIS will be evaluated. 
First, the sampling setting and participants will be 
discussed. This will also include a brief background about 
the CHCS. Next, the measurement scales used in this study will 
be presented along with the level of confidence used in the 
statistical analysis. 

Validation of the short-form UIS instrument will take a 


series of progressive steps in order to answer the research 


questions. Although the three sampling sites are relatively 
the same (li.e., medium-sized naval hospitals), and could be 
assumed to represent a relatively homogenous’ sampling 
population, oneway analysis-of-variance (ANOVA) will be 


performed on the sampling demographics. Using the three site 
combined data, common factor analysis methods will be 


conducted. Common factor analysis is considered one of the 
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most powerful methods of construct validation as it allows 
the examination of the underlying structure of the overall 
measure (3][27]. Internal consistency or reliability testing 
of each factor as well as other statistical measures of 
overall factor structure "goodness of fit" will be used. 
Equivalence testing is conducted to test the invariance of 
the factor structure and loading across the three sites, and 
is important to demonstrate that there is no instrument bias. 
Lastly, responses from user interviews, and comments offered 
on the completed instruments will be used to look at 
convergent validity. 

Evaluation of the findings from the short-form UIS 


instrument as mentioned above will rely on the outcome from 


validation. Presentation of the data will use statistical 
testing techniques to demonstrate differences in means 
between the defined user groups and sites. Correlation 


testing between length of time user use of the CHCS and UIS 
will also be evaluated. Correlation statistical methods will 
be used and tested for significance. Additionally, trend 


analysis will be conducted and presented. 


B. SAMPLE AND DATA COLLECTION 
1. Background of the Composite Health Care System (CHCS) 
CHCS is ae state-of-the-art, integrated, medical 
information system the Department of Defense (DoD) is testing 
for implementation at its medical treatment facilities. On 
the leading edge of technology and beyond the capabilities of 
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systems commercially available, CHCS is designed to improve 
the timeliness, availability, and quality of patient-care 
data. ic will replace manual and automated information 
system now supporting DoD medical treatment facilities. At 
individual hospitals, it will integrate the functional work 
centers of inpatient and outpatient care facilities, patient 
administration, patient appointment and scheduling, nursing, 
laboratory, pharmacy, radiology, and clinical dietetics. 
CHCS is intended to provide physicians with immediate access 
to patient medical records. [28] 

The approval of the Mission Elements Need Statement 
(MENS) in February 1979 affirmed the DoD's goal of providing 
integrated information support via a computerized system for 
its medical treatment facilities. A two-stage competitive 
acguisition process was undertaken to solicit industry's best 
solution to the needs of DoD medical treatment facilities. 
On September 10, 1986, stage I of the contract was awarded to 
four competing vendors. Each vendor was to design, develop, 
and implement levels I and II of CHCS (essentially outpatient 
services) at predesignated test sites. Stage II consisted of 
an extensive evaluation that included an extended benchmark 
test in order to aid in the selection of one of these 
vendors. Based on the evaluation, Science Applications 
International Corporation (SAIC) of San Diego, California was 
selected as the source to further develop and deploy CHCS to 


12 operational testing and evaluation (OT&E or beta test) 
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sites. Of the 12 OT&E sites, three are Navy, five are Arny, 
and four are Air Force medical treatment facilities. This 
group of OT&E sites represented hospitals with the number of 
beds ranging from 40 to 886, and outpatient clinic visits 
ranging from 247,285 to 1,573,369 per year. DoD estimates 
CHCS life-cycle costs (the expected life of the system is 
through fiscal year 2002) for full deployment to 767 medical 
treatment facilities at $1.6 billion.[1][28][29] 

CHCS iS a menu-driven, networked system, safeguarded 
via a password access hierarchal assignment according to the 
appropriate level of communication needs of the user. That is 
to say, that physicians as the focal point of health care and 
treatment are given the most authority to input and extract 
information from the system. CHCS also provides electronic 
mail communication and the system is intended to interface 
with other Hospital System Program Office (HSPO) and DoD 
initiatives Tyg olkivyelijers ye food service, medical logistics, 
Defense Enrollment Eligibility Reporting System (DEERS), 
Medical Expense and Performance Reporting System (MEPRS), 
service-specific administrative systems, tactical automation 
systems, and National Disaster Medical System and Veterans 
Administration Systems. [29] 

2. Sample and Data Collection 

The three naval hospital OT&E sites (Charleston, South 

Carolina: Jacksonville, Florida; and Camp Lejeune, North 


Carolina) were evaluated in this study. Hurd's [1] data from 
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the Naval Hospital Charleston, South Carolina, was used in 
this study in lieu of re-sampling. The three sites are 
essentially the same size (medium-sized naval hospitals) with 
the number of hospital beds ranging from 170 to 184, and 
outpatient clinic visits ranging from 360,000 to about 570,000 
per year [28]. No complete inpatient module implementation 
had occurred at any of these three naval hospitals at the 
time of data collection for study. Therefore, the nursing 
unit module was not evaluated. Additionally, the clinical 
dietetics module was not implemented. The modules that were 
implemented were: patient administration (PAD), patient 
appointment and scheduling (PAS), pharmacy (PHR), laboratory 
(LAB), and radiology (RAD). The PAD module was still being 
run in parallel with the AQCESS system, and did not have the 
cash collections component (MSA) on-line. The LAB module did 
not have the blood transfusion service component on line. 
The short-form UIS instrument used was the same as 
that used by Hurd [1] due to the incorporation of his data 
from the Naval Hospital Charleston, South Carolina, into this 
study (refer to Appendix A). However, questions 8 and 10 were 
given further clarification using the definitions offered in 
the original study by Bailey and Pearson [18]. It was felt 
that there could be misunderstandings or ambiguity in 
interpretation by the subjects without this clarification, 


and would not jeopardize the instrument's integrity due to 


ly 


both of these questions are expected to factor within the 
same factor (Information Product Output). 

A data collection set consisted of a cover letter, the 
short-form UIS instrument, and an addressed envelope. The 
cover letter informed the subjects that their responses would 
be treated in complete confidentiality, and where to direct 
their sealed envelope response at each hospital. The Naval 
Hospital Camp Lejeune, North Carolina, and the Naval Hospital 
Jacksonville, Florida, were the two other Composite Health 
Care System (CHCS) facilities sampled. Prior to sending out 
the data collection sets to the Naval Hospital Camp Lejeune 
and the Naval Hospital Jacksonville, each facility's CHCS 
Project Officer was contacted to obtain their support and 
assistance inthis study. Additionally, command notification 
and permission was obtained at each facility. 

Each CHCS Project officer was sent 250 data collection 
sets. Additionally, general guidance for dissemination at 
each facility was provided to each CHCS Project Officer to 
ensure random sampling of all CHCS users in the outpatient 
areas (l.e., physicians, clinics, laboratory, radiology, 
pharmacy, and administrative departments). Returned 
instruments and Hurd's [1] Naval Hospital Charleston data 
were coded for responses’ and categorized into three 
functional groups: physicians, medical Support, and 
administrative support. These groupings were different 


from Hurd's [1] study, however, these groupings 
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better represent the different user groups in terms of the 
function provided to health care, and therefore use of the 
CHCS. The physician group is self explanatory. The medical 
support group consisted Oe nurses, pharmacists, 
technologists, other health care professionals, technicians, 
and hospital corpsman not working in an administrative 
department. The administrative support group consisted of: 
all administrators, secretarial and clerical persons 
regardless of department assigned to, and all other persons 
regardless of profession assigned to an administrative 
department. Education, sex, use of other computer systems, 
and use of other health care information system demographic 
information were given numerical values (i1.e., male=l, 
female=2, no=0, yes=1, high school graduate=0, etc.) in order 
to allow comparative analysis using these attributes, or 
enable ease of database tabulation. 

All responses were inputted using an integrated 
modular software package (Enable/OA). This package allows the 
researcher to build a database of responses and ease of data 
retrieval and importation of this data into a spreadsheet 
format for statistical analysis. Each of the questions 
(variables), factors, and overall satisfaction scores were 
computed as described by Baroudi and Orlikowski [2] (refer to 


Chapter ITI). 


iS, 


C. MEASUREMENT SCALES 

In psychological or behavioral studies, there appears to 
be some confusion as to the legitimacy of uSing particular 
classes of mathematical procedures [8][27]. Specifically, the 
use of parametric statistical procedures verses nonparametric 
procedures with measures of psychological attributes. 
Parametric statistical procedures have more statistical power 
than nonparametric procedures, yet require at a minimum that 
there exists interval scales. According to Nunnally, an 
interval scale is "1) the rank-ordering of objects is known 
with respect to an attribute and 2) it is known how far apart 
the objects are from another with respect to the attribute, 
but 3) no information iS available about the absolute 
magnitude of the attribute for any object" [27, p. 16]. 
Scaling models such as the seven interval Likert-type scaling 
model used with the short-form UIS instrument, are applied by 
the researcher to what appears to be ordered categories (or 
ordinal scales) to the subjects, to convert the data into 
interval scales. Nunnally [27] strongly believes that it is 
permissible to take seriously the intervals among scores in 
performing analyses of attitude such as that used with the 
short-form UIS instrument. It is beyond the scope of this 
paper to fully explain the rationale behind these arguments, 
and it is recommended that readers review Nunnally [27]. 

In this study, parametric procedures will be used, 


however, nonparametric procedures will be used in those cases 
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where testing failed to support the underlying assumptions 
for parametric procedures (specially, in the oneway ANOVA 
procedure). All testing was conducted at a confidence level 


of 95% or alpha = 0.05. 


D. VALIDATION OF THE INSTRUMENT 
1. Homogeneity of the Sample Data Sets 

Oneway analysis-of-variance (ANOVA) using a PC version 
of MINITAB statistical program [30] and macro programs 
provided by Zehna [31] were performed on the following 
demographic attributes: education, age, gender, length of 
time (in months) of CHCS use, use of other computer systen, 
and use of other health care information systems to ascertain 
homogeneity of the sample data sets. ANOVA methods have been 
developed to test for differences between the means of 
several groups. In this study, ANOVA procedures were applied 
to the three subpopulations: Charleston, Camp Lejeune, and 
Jacksonville. Where significant differences in means 
occurred, the Scheffe multiple comparison testing was 
conducted a posteriori. NOCH ena, a posteriori testing 
for normality and homogeneity of variance was conducted. 
Normality was tested using MINITAB's option for computing and 
storing fitted and residual values. Applying the NSCORE 


function to compute the normal scores of the residuals and 


then compute the correlation of the normal score with 
residuals approximates a normal distribution if the 
correlation is large (1.e., the closer to 1.0 the better) 
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[30]. The Hartley's Fmax test was used for homogeneity of 
variance. 
2. Common Factor Analysis 

The common factor-analytic model is different from 
principal components analysis in that it makes a distinction 
between common and specific parts of variables. In principal 
components analysis, the goal is to construct linear 
combinations of the original variables that account for a 


large part of the total variation. That is to say, the 


unobserved factors (latent variables) are expressed as 
functions of the observable variables, and 1S variance 
oriented, and without an error’ term. The common 


factor-analytic model, on the other hand, expresses each 
observable variable in terms of unobservable common factor and 
a unique factor, and 1S covariance oriented. The common 
variance of a variable is also called the communality of the 
variable. The communality of a variable is the portion of a 
variable's total variance that is accounted for by the common 
factors. With the principal components analysis there is no 
error term. Conceptually, the absence of an error term 
implies that the observable variables are measured without 
error and that the unobservable latent principal component is 
a perfect linear combination of its measures or are formative 
indicators of the unobservable factor. Whereas, common 
factor analysis is reflective in that the indicators subject 


to measurement error are a function of wunobservables. 


Ze 


Instrument constructs to assess attitude are typically viewed 
as underlying factors that give rise to something that is 
observed, and therefore their indicators (i.e., the observed 
variables) should be viewed as reflective, hence the use of 
the common factor-analytic model. Common factor-analytic 
techniques can better serve the functions of searching the 
data for qualitative and quantitative distinctions and, 
especially testing a priori hypotheses and statistical 
testing criterion. [32] 

The maximum-likelihood common factor analysis is 
preferred due to its ability to test hypotheses about the 
number of common factors. There are two different data 
analysis contexts: exploratory and confirmatory. Exploratory 
factor analysis iS Simply searching for a common structure 
underlying the data without having any theoretical hypothesis 
in mind. Whereas, confirmatory factor analysis there exists 
some prior theoretical information on the common structure 
underlying the data and one wishes to confirm or negate the 
hypothesized structure. [32][33} 

The rotation process of factor analysis pattern matrix 
provides a clearer delineation oof the pattern  0of 
relationships. Rotation options allow for a simple factor 
solution to become clearer. There are two methods in which 
the factor axes can be rotated. Orthogonal rotation 
preserves the original orientation between the factors so 


that they are still perpendicular after rotation. Whereas, 
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oblique rotation, the factor axes can be rotated 
independently. Varimax orthogonal rotation is one of the 
most popular rotation techniques used. Varimax rotation 
Spreads variance evenly among factors while maintaining the 
original orientation between the factors so that they are 
Still perpendicular after rotation. The procedure seeks to 
rotate factors so that the variation of the squared factor 
loadings for a given factor is made large. [32-34] 
a. Exploratory Factor Analysis 

divelabie strat lL IA exploratory factor analysis was 
undertaken using the SAS maximum-likelihood factor analysis 
procedure [33] on the combined data. Multivariate normality 
was assumed in conducting the exploratory factor analysis. 
Cattell's scree test was performed for determining the 
approximate number of factors to extract. The Cattell's 
scree test is simply a visual determination of the point 
where the factors curve above an approximate straight line 
made from the bottom roots [34]. SAS has the capability of 
computing the Kaiser-Meyer-Olkin (MSA option) measure of 
sampling adequacy [33]. The MSA is a summary of the extent 
to which the variables belong together and are thus 
appropriate for factor analysis [34]. MSA's greater than 0.8 
can be considered good [33][34]. The Schwartz's Bayesian 
Criterion is used to determine the best number of factors to 
be extracted using the maximum likelihood factor analysis 


procedure. The number of factors that yields the smallest 
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value for the Schwartz's Bayesian Criterion is considered the 
best extraction [33]. The Schwartz's Bayesian Criterion 
according to the SAS user guide seems to be less inclined to 
include trivial factors than either the Akaike's Information 
Criterion or the chi-Square test [33]. In the literature, 
there have been problems reported in using the chi-square test 
due to its susceptibility to sample size [35-38]. SAS also 
provides the Tucker and Lewis's Reliability Coefficient 
automatically for maximum-likelihood factor analysis 
procedure. The closer the Tucker and Lewis's Reliability 


meetticient is tO 1.0 the better the factor solution fit. 


SAS also automatically computes the squared canonical 
correlation (which is the same as squared multiple 
correlations) for maximum-likelihood factor analysis 


procedure. The squared multiple correlation (SMC) for each 
variable is the relative variance in that variable which is 
accounted for by the overall factor solution jointly [33][39]. 
Basically, the SMC represents the lower bound of reliability 
that each variable contributes to the overall factor 
structure. The maximum-likelihood factor analysis procedure 
because it iS an iterative process using SMCs for initial 
estimates (using SAS) is susceptible to quasi- or 
ultra-Heywood cases. It is beyond the scope here to discuss 
these anomalies, however, SAS has a Heywood option which sets 
to 1 any communality greater than 1, allowing iterations to 


proceed until convergence criterion is met [33]. The Varimax 
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rotation option was used in the SAS procedure program. Lastly, 
the data was standardized using the SAS procedure STANDARD and 
retested using the maximum-likelihood procedure as above. 
Appendix B provides the SAS commands used for the exploratory 
factor analysis. 
b. Confirmatory Factor Analysis 

After obtaining the optimal factor structure 
solution via exploratory factor analysis, confirmatory factor 
analysis was conducted. This time multivariate normality 
was not assumed and the observed variables were analyzed for 
goodness of fit to the optimal exploratory factor analysis 
model using Joreskog and Sorbom's LISREL 7 (Linear Structural 
Relations) computer program. The LISREL model can be viewed 
in terms of a confirmatory factor analytic model [32]. 
Joreskog and Sorbom [39] in their LISREL 7 manual provide 
testing cases for non-normality where the observed variables 
are on interval scales using Weighted Least Squares analysis. 
Using Joreskog and Sorbom's [40] PRELIS program, the raw data 
is converted and saved as a polychoric correlation matrix and 
an asymptotic covariance matrix to be used in the confirmatory 
factor analysis [39][40]. Appendix C provides the PRELIS 
commands used and the LISREL commands used for confirmatory 
factor analysis. In addition to the Total Coefficient of 
Determination (TCD), Goodness of Fit Index (GFI), Adjusted 
Goodness of Fit Index (AGFI), and Root Mean Square Residual 


(RMR), the output will provide the Standard Errors (SE), and 
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t-values (TV). LISREL's t-values or critical ratios when 


larger than two are normally judged to be significantly 


different from zero, and therefore indicating a true 
parameter for loading in that factor [3942 The TCD is a 
measure of how well the variables jointly serve as 


measurement instruments for the overall factor structure. The 
closer to 1.0 the TCD, GFI, and AGFI are, the better the model 
fits the data. The RMR and SE's should all be very small to 
indicate overall good fit of the data. There is some debate 
about the use of the GFI. Mulaik et al. [35], in a recent 
evaluation recommends the use of GFI when one has samples at 
least 200 in size, and of course when the conditions for that 
method are satisfied. 
3. Reliability (Internal Consistency) Testing 

Internal consistency was tested for each factor using 
Cronbach's Alpha reliability coefficient option in SAS's 
correlation procedure [41]. Cronbach's Alpha is based on the 
average correlation of items within a factor (or test). It 
represents the correlation between this factor (or test) and 


all other possible factors (or tests) containing the same 


number of items, which could be constructed from a 
hypothetical universe of items that measure the 
characteristic of interest (1.e., the factor). It also sets 


an upper limit to the reliability of the fACtOr. hetac 
proves to be very low, either the factor has too few items or 


the items have very little in common [27]. According’ to 
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Nunnally (27], reliability coefficients of .70 orf higher wae 
suffice in the early stages of research, and for basic 
research, efforts to increase much beyond .80 is often a 
waste of time and funds. Appendix D provides the SAS 
commands used to obtain the Cronbach Alpha's for each factor. 
4. Testing for Measurement and Structure Invariance 

Having conducted common factor analysis on the data as 
a whole, the next verification is to test that the factor 
structure and loading are the same for each of the three 
subpopulations. In addition, before conducting ANOVA 
testing of the three subpopulation location's for evaluating 
the difference in means, it is important to confirm that the 
measurement and the structure of the instrument designed to 
measure attitudes are equivalent across the subpopulations 
(37][38][42]. Joreskog and Sorbom [39] provide a methodology 
to analyze data from samples simultaneously using their 
LISREL models. They outline a series of tests to be 
conducted to confirm measurement and structural invariance. 
The first test (hypothesis A) is an overall test of the 
equality of covariance matrices (or structures) across the 
three subpopulations. Failure to reject the null hypothesis 
(i.e., covariance matrices are equal) is statistical evidence 
that the groups can be treated as one. 

The next series of testing consists of a model in 
which certain parameters are constrained to be equal across 


the subpopulations is compared with a less restrictive model 
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where these same parameters are free to take on any value 
[(37][39]. In each of these tests, at least one of the scales 
or items making up each of the factors must be fixed to 1.0. 
In this study, the highest loading item (from previous factor 
analysis above) was fixed to 1.0. There is no guidance in 
the literature as to which item to fix, and to iteratively 
fix the various items in each factor is problematic. MThe 
choice uses that item which strongly loads into its 
respective factor, and therefore setting it to 1.0 seemed 
appropriate. The next test (hypothesis B) tests that there 
are four factors in all three subpopulations with a factor 
pattern of: Factor A consists of questions 1, 6, 11; Factor 
B consists of questions 2, 12; Factor C consists of questions 
7, 8, 9, 10, 13; and Factor D consists of questions 3, 4, and 
5. Assuming failure to reject hypothesis B, hypothesis C 
tests that there is invariance in factor loadings (lambda x) 
across the subpopulations. Assuming failure to reject 
hypothesis C, hypothesis D tests that there is invariance in 
Ehe error/uniqueness (theta) across the subpopulations. 
Lastly, assuming failure to reject hypothesis D, hypothesis 
E tests that the factor variances and covariances (phi) are 
invariant across the three subpopulations. 

For hypothesis B, subpopulations 2 and 3 are specified 
to have the same pattern and the same starting values as 
subpopulation 1 (LX=PS command on the LISREL 7 model input 


line). In hypothesis C, subpopulations 2 and 3 are specified 
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to be invariant for factor loading from subpopulation 1 
(LX=IN command on the LISREL 7 model input line). Hypothesis 
D additionally constrains the theta matrices to be invariant 
(TD=IN command on the LISREL 7 model input line). Hypothesis 
E additionally constrains the phi matrices to be invariant 
(PH=IN command on the LISREL 7 model input line). Appendix 
E provides the PRELIS and LISREL commands’ used for each of 
the hypothesis testing. 

The LISREL 7 computer program output provides the GFI 
and RMR for each subpopulation. The chi-square measure 
provided with the last subpopulation is the measure of the 
overall fit of the three subpopulations. Alternative indices 
used to help evaluate LISREL models in multiple sample 
analysis where the chi-square measure and degrees of freedom 
are reported as summed values from the multi-sample testing 
(as in this testing) are: the chi-square to the degrees of 
freedom ratio, and the chi-square likelihood ratio tests. 
The chi-square/degrees of freedom ratio is distributed as a 
t-statistic so that anything greater than 1.96 (in this 
instance, where n = 340, and alpha = 0.05) is significant 
[38]. The chi-square likelihood ratio (LR), also described as 
the chi-square difference test, is utilized where restricted 
nested models are used as in this case where hypotheses C, D, 
and E are restricted nested models of hypothesis B. The LR 
test is calculated by taking the difference in the chi-square 


estimators for the restricted and unrestricted models and the 


30 


difference in degrees of freedom (df) for the two models and 
reporting as a chi-square/df ratio [37)[38])[43]. 
5. Post Survey Interviews (Convergent Validity) 

Convergent validity is the extent that a measure is 
correlated or "agrees" with other measures of the same 
@onetecruct [3]. Interviews were conducted with randomly 
selected members of the user groups at the Naval Hospitals 
Camp Lejeune, and Jacksonville. The interviews were 
conducted to assess users overall satisfaction with the 
system for comparison with the instrument's results. 
Additionally, the interviews were used to gain comments about 
the system, and the short-form UIS instrument used in this 
study. Subjects interviewed were assured that their 


responses would be kept confidential. 


E. EVALUATION OF THE INSTRUMENT!'S DATA 
1. Testing Differences in Means 

Oneway analysis-of-variance (ANOVA) testing was 
conducted as described above for testing homogeneity of 
sample data sets. First, ai combined three hospital data 
set was used to test the difference in means between the 
three user groups (physicians, medical support, and 
administrative support) for each of the 13 questions, overall 
score, and factors. Using the combined data set, each user 
group was tested for differences in means between the three 


location sites (i1i.e., Charleston vs Camp Lejeune vs 
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Jacksonville). Lastly, each hospital's data was tested for 
differences in means between the three user groups. 

Where significant differences in means occurred (i.e., 
P value less than 0.05), the Scheffe multiple comparison 
testing was conducted a posteriori. Additionally, a 
posteriori testing for normality and homogeneity of variance 
was conducted. Normality was tested using MINITAB's option 
for computing and storing fitted and residual values when 
performing oneway ANOVA procedure. Applying the NSCORE 
function to compute the normal scores of the residuals and 
then computing the correlation of the normal score with 


residuals approximates a normal distribution if the 


correlation is large (i.e., the closer to 1.0 the better) 
Estos The Hartley's Fmax test was used for homogeneity of 
variance. 


Nonparametric Kruskal-Wallis ANOVA testing of sample 


medians was conducted in those situations where parametric 


ANOVA testing assumptions were violated (i.e., lack of 
normality and/or homogeneity of variance). Using a PC 
version oof MINITAB statistical program (30 te the 


Kruskal-Wallis test statistic H and P values were calculated 
and adjusted for ties in responses. Where Significant 
differences in medians occurred, the $MINITAB [30 
nonparametric Mann-Whitney two-sample median procedure was 
performed in pairwise comparisons to identify individual 


Significant differences. 
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2. Time of Use Correlation Testing 

The assumption held is that as the length of time of 
use of the system increased, the user's level of satisfaction 
would increase. The CHCS is a mandatory use system as 
opposed to an optional use system. As such, medical 
personnel must use the CHCS to accomplish their work (in 
those areas where respective CHCS modules have been 
installed). 

Correlation measures the degree of association between 
two variables. The range of correlation strength can be from 
-1.0 (perfect negative correlation) to +1.0 (perfect positive 
correlation), with zero meaning no correlation. The term 
"negative" used here with correlation denotes that as one 
variable increases the other variable decreases. The term 
"positive" used with correlation denotes that as one variable 
increases the other variable increases. The independent 
variable used in this study was time of use of the CHCS (in 
terms of months). The dependent variable to test against 
were each factor's mean scores. The Pearson's’ sample 
correlation coefficient (r) was obtained for each comparison 
using MINITAB. It should be noted that rho and its estimate 
r are both symmetric so that the two variables to be 
correlated can be interchanged without changing the value. 
It is because of this symmetry that no cause and effect 
statement may be made, rather just the strength of 


association or relationship between the two variables [31]. 
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The Pearson's sample correlation coefficient was tested for 
Significance using a macro program provided by Zehna [31] for 
MINITAB. The testing of the sample correlation coefficient 
(r) used one-tail hypothesis testing where Hl: rho > 0 if r 
waS positive or Hl: rho < 0 if r was negative to obtain the 
appropriate P value. A P value of less than 0.05 indicates 
that the null hypothesis of no correlation may be rejected. 
Correlations were performed on the combined three hospital 
data. 
a. Trend Analysis 
Hurd [1] found that none of the work groups 
demonstrated any high correlation between time of use and the 
level of satisfaction. However, he used six month time series 
intervals to look for possible trends (negative or positive) 
between the time of use of the CHCS and the overall UIS summed 
score. He found at the Naval Hospital Charleston, that 
physicians and administrative support tended to exhibit a 
positive trend-line, whereas, the ancillary group (which is 
part of the Medical Support group in this study) tended to 
exhibit a negative trend-line for overall satisfaction. 
In this study, the trend analysis performed by 
Hurd [1] was replicated. Trend analysis of the mean score in 
six month intervals for each factor was conducted. The 
number of individuals involved in each six month interval and 


the percentage of the whole were tabulated to provide clarity 
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as to the weighing of the results. The trend analysis was 


performed on the combined three hospital data. 
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IV. DESCRIPTIVE FINDINGS 


A. DEMOGRAPHIC FINDINGS BY LOCATION AND AS A GROUP 
Of the two Naval hospitals surveyed in this study, 121 
usable instruments were obtained from the Naval Hospital Camp 
Lejeune, and 118 usable instruments were obtained from the 
Naval Hospital Jacksonville. The response rates for the 
Naval Hospitals Camp Lejeune and Jacksonville were 48% and 
47%, respectively. These rates of response were similar to 
that obtained by Hurd [1] in his study where he had 101 
respondents and a response rate of 56%. Hurd's data from the 
Naval Hospital Charleston is used in this study. Appendix F 
contains a summary of the demographics by location and as a 
group (the three hospitals combined). 
1. Age 
Hurd's [1] data revealed an average age of respondents 
from the Naval Hospital Charleston to be 32 years, with a 
range in years from 19 to 56. The average age of respondents 
from the Naval Hospital Camp Lejeune was 33 years, with a 
range in years from 18 to 61. The average age of respondents 
from the Naval Hospital Jacksonville was 32 years, with a 
range in years from 19 to 56. The combined group had an 


average age of 32 years with a range in years from 18 to 61. 
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2. Gender 
The gender of the respondents from the Naval Hospital 
Charleston were 57 (56%) male and 44 (44%) female. The gender 
of the respondents from the Naval Hospital Camp Lejeune were 
poe (60%) male and 48 (40%) female. The gender of the 
respondents from the Naval Hospital Jacksonville were 83 
(70%) male and 35 (30%) female. The gender split in the 
combined group of respondents were 213 (63%) male and 127 
(37%) female. 
3. Hospital Departments 
Only outpatient departments were sampled. Work areas 
reported were in one of the following department types: 
clinic, administration, laboratory, pharmacy, or radiology. 
Figure 4.1 depicts the individual and combined hospital 


department types and percentages. 


oy 


DEPARTMENT WORK CENTERS 


- “J \ 4 y 
7 i m Whe F 
g A 

rt ‘; 
A he 
s 
iN - é 

' *, p 


NH Charleston, SC 


= 


NH Camp Lejeune, NC NH Jacksonville, FL 


WY Administration Ws Clinics Laboratory Y Pharmacy = Radiology 





the 


Figure 4.1. Department Work Centers 


4. Job Descriptions 


Job descriptions reported were categorized into one of 


following types: 


Physician - a Medical Corps Officer or civilian 
equivalent. 


Nurse - a Nurse Corps Officer or civilian equivalent. 


Health Professional - a Medical Service Corps Officer 
(Allied Science) or civilian equivalent. 


Administrator - a Medical Service Corps Officer 
(Health Care Administration) or civilian equivalent. 


Technician - a Hospital Corpsman with a medical 
technician rating or civilian equivalent. 


Corpsman - a Hospital Corpsman without a technical 
mace WMG 


Clerical - a person performing secretarial or clerical 
functions. 
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Figure 4.2 depicts the individual and combined 


hospital job description types and percentages. 
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Figure 4.2. Job Description 


5. Functional User Groups 

Based on the person's work department and job 
description types they were categorized into three functional 
work groups: Physicians, Medical Support, and Administrative 
Sepport. The Physician group is self explanatory. The 
Medical Support group consisted of: nurses, health 
professionals, technicians, and corpsman not working in an 
administrative department. The Administrative Support group 
consisted of: all administrators, and clerical persons 
regardless of department assigned to, and all other persons 


regardless of profession assigned to an administrative 
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department. Figure 4.3 depicts the individual and combined 


hospital user group types and percentages. 
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Figure 4.3. User Groups 


6. Level of Education 

As expected, the participants working within a health 
care setting possess a high level of education. Looking at 
the combined hospital data, Figure 4.4, almost 90% of the 
respondents have some college or higher educational 
experience. The median education level of the respondents 
from the Naval Hospital Charleston and from the Naval 
Hospital Camp Lejeune was "some college." The median 
education level of the respondents from the Naval Hospital 


Jacksonville was "bachelor degree." The median education 
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level of the combined group was "Some college." Figure 4.4 
depicts the individual and combined hospital level of 


education and percentages. 
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Figure 4.4. Education Level 





7. Computer Use 

Computer use is not an unfamiliar task with this group 
of participants. 80% of all the participants have used other 
computer systems of some sort. Of these users, nearly three 
quarters of them (74%) have used other health care information 
system(s). When comparing all respondents, 59% have used 
other health care information system (HCIS). The respondents 
from the Naval Hospital Charleston reported a 80% use of 
other computer systems; of those users, 71% have used a HCIS 


and of all respondents, 58% have used a HcIS. The 
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respondents from the Naval Hospital Camp Lejeune reported a 
76% use of other computer systems; of those users, 68% have 
used a HCIS and of all respondents, 52% have used a HCIS. 
The respondents from the Naval Hospital Jacksonville reported 
a 86% use of other computer systems; of those users, 76% have 
used a HCIS and of all respondents, 66% have used a HCIS. 
8. Length of Time of CHCS Use 

Of all respondents, the average length of time in 
months of use of the CHCS was 12.3 months. The overall time 
of use ranged from one month to 36 months. The respondents 
from the Naval Hospital Charleston reported an average time 
of use of 8.6 months, with a range of one to 19 months. The 
respondents from the Naval Hospital Camp Lejeune reported an 
average time of use of 12.5 months, with a range of one to 36 
months. The respondents from the Naval Hospital Jacksonville 
reported an average time of use of 15.3 months, with a range 


of one to 32 months. 


B. VALIDATION OF THE INSTRUMENT 
1. Homogeneity of the Sample Data Sets 
The oneway ANOVA testing results are presented in 
Table 4.1a. Significant differences in means are denoted by 
underlining. Significant differences in subpopulation means 
were noted in the education and the time of use of CHCS user 


attributes. 
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TABLE 34 us 
USER ATTRIBUTES ANOVA TESTING 


——eEq]ESSEEoeESeEEEEyyEE———eEEeeEEe—E—EELLLES SS ESE LS — LESSEE Se 
NSCORES HARTLEY 
ATTRIBUTE F VALUE P VALUE CORRELATION FMAX(1) 
AGE O 16 0.852 0 98 1.59 
GENDER 2 48 0.086 0 8&8 1 18 





COMPUTER SYSTEM EXPERIENCE 2 18 0.115 0.81 1.55 


HOSPITAL INFORMATION SYSTEM EXPERIENCE 2.45 0.088 0 89 1.41 
EDUCATION 4.09 0.018 0.93 Us) 





CT) Hartley's Fmaxc3 449) aporoximate critical value =1.803 alpha = 0.05 
The Scheffe interval comparison testing for the 
education attribute is presented in Table 4.1b. Education 
was tested and found to have no significant correlation with 
the overall satisfaction, as well as with each of the four 
factors found when factor analysis was performed. Time of 
use of the CHCS system is discussed later in this paper. 


TABLE 4.1B 
SCHEFFE MULTIPLE COMPARISON TESTING OF EDUCATION ATTRIBUTE 


nee a ll le i MM A Li! 
SITE LOCATION COMPARISONS 


CHARLESTON/ CHARLESTON/ CAMP LEJEUNE/ 
ages UTE CAMP LEJEUNE JACKSONVILLE JACKSONVILLE 
EDUCATION NS(1) NS(1) SCe) 


NOTES: 

(1) NS = nonsignificant: no difference between the means 

(2) S = significant difference between the means 
PET 


2. Exploratory Factor Analysis 
The combined data set (n=340) was used to perform the 
exploratory factor analysis procedure. The Cattell Scree Test 
suggested a maximum of five factors may possibly exist. The 
SAS maximum likelihood factor analysis procedure was written 
to sequentially perform a one-factor solution through a 


five-factor solution. The Kaiser's Measure of Sampling 


Adequacy (MSA) had a value of 0.89 indicating a very good fit 
of the data for factor analysis. The maximum likelihood 
factor analysis procedure in SAS uses squared multiple 
correlations (SMC) as its initial starting estimate. The SMC 
is the lower bound for the reliability of each variable. The 
SAS squared canonical correlations (SCC) reported for each 
factor is the lower bound of reliability for that factor (from 
the variables that make up that factor) [33]. The one-factor 
solution made up of all 13 questions had a SCC value of 0.90. 

The Schwarz's Bayesian Criterion (SBC) value kept 
decreasing for the two-factor and the three-factor solutions. 
This indicated that a greater than three-factor solution was 
optimal. The three-factor solution had a SBC of 219 and a 
Tucker and Lewis's Reliability Coefficient of 0.91. At the 
four-factor solution, the SBC reached its lowest value of 
211, and rose to the value of 217 at the five-factor 
solution. Table 4.2 shows the optimal four-factor solution. 
The Tucker and Lewis's Reliability Coefficient was 0.95 for 
the four-factor solution. 

Factor A is made up of questions 1, 6, and 11, and 
represents the local Management Information Department (MID) 
staff and services. Factor B is made up of questions 2, and 
12, and represents the contractor's services. Factor C is 


made up of questions 7, 8, 9, 10, and 13, and represents the 
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information product output. Factor D is made up of questions 
3, 4, and 5, and represents user knowledge and involvement. 
3. Confirmatory Factor Analysis 

Paramount to the maximum likelihood factor analysis 
procedure is that the data be multivariate normal. There is 
no easily defined test for multivariate normality that could 
be found in the literature. Instead, the Joreskog and 
Borbom's [39] LISREL model for analysis of non-normal 
variables was used. The Total Coefficient of Determination 
(TCD) for the variables was 0.997 indicating a very good fit 
to the four-factor solution. Other goodness of fit indices 
used supported the four-factor solution. The Goodness of Fit 
Index (GFI) was 0.985, and the Adjusted Goodness of Fit Index 
(AGFI) was 0.977. 

The Root Mean Square Residual (RMR) was 0.041, which 
also Supports the goodness of fit of the four-factor 


solution. Additionally, Standard Errors (SE) and t-values 


fefskEL's critical ratios) were performed for each of 
variable's loading into their respective factor. The SE's 
were all low (<.04), and the t-values were all large (>20) 


for each of variable factor loadings, which further supports 
the goodness of fit of the four-factor solution. 
4. Reliability (Internal Consistency) Testing 
Factor A 1s made up of questions 1, 6, and 11, and 
represents the local Management Information Department (MID) 


staff and services. Factor B 1s made up of questions 2, and 
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12, and represents the contractor's services. Factor C is 
made up of questions 7, 8, 9, 10, and 13, and represents the 
information product. Factor D is made up of questions 3, 4, 
and 5, and represents knowledge and involvement. 

The Cronbach's alpha for Factor A was 0.89; for Factor 


B was 0.68; for Factor C was 0.87; and for Factor D was 0.75. 


TABLE 4.2 
MAXIMUM LIKELIHOOD FACTOR ANALYSIS WITH VARIMAX ROTATION 


FACTOR B 


FACTOR A 
FACTOR C FACTOR O 
MIO STAFF ANO CONTRACTOR 
QUESTIONS/VARIABLES INFORMAT LON KNOWLEDGE AND SMC 
SERVICES SERVICES Aptis Neer 

1 RELATIONSHIP WITH MIO STAFF 0.75 0.18 0 18 0.20 0.61 
2. eer OF REQUESTS FOR Ne 0.58 0 19 0.16 0.36 
3. OEGREE OF TRAINING PROVIOEO 0027 0.34 0 16 0.48 0.43 
4. eoenee UNDERSTANDING OF 0 16 0.11 0 10 0.79 0 37 
5. USER'S FEELING OF 

C PERICIOR ETON 0.25 032 0.30 0.51 0.48 
6. ATTITUDES OF MIO STAFF 0.79 0.14 0 20 0.15 0.63 
7. RELIABILITY OF OUTPUT 0 16 0 25 0.77 0.07 0.61 
8 RELEVANCY OF OUTPUT 0 23 0.32 0.61 0.15 0.54 
9. ACCURACY OF OUTPUT 0 20 0 15 0.75 0.09 0156 
10 PRECISION OF OUTPUT 0.12 0.05 0.71 0 19 0.48 
11 COMMUNICATING WITH MIO STAFF 0.83 0 14 0 22 0 19 0.69 
12. TIME REQUIREO FOR NEW 

LES i cs 0.17 0.69 0 23 0.19 0 45 
13. COMPLETENESS OF OUTPUT 0.20 0 44 0.59 0.10 O57 
Eigenvalue 4 10 2 30 15 90 1 30 -- 
cumulative percent 17% 26% 94% 100% -- 
SCC 0 80 0.70 0.94 0.56 aS 
cronbach’s Alpha oO 89 0.68 O 87 075 -- 


Schwarz Bayesian Criterion = 211 


Tucker and Lewts Reliability Coefficrent = 0.95 
SS SSS SSS SEES =—EEst 


5. Testing for Measurement and Structural Invariance 
The use of Joreskog and Sorbom's LISREL model to test 
for measurement and structural invariance revealed that the 
three subpopulations are equivalent for their responses and 
the four-factor data reduction. Hypothesis A was that the 


covariance structure across the three subpopulations is 
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invariant. Hypothesis B was that the number of factors of 
the factor structure is the same across the three 
subpopulations is invariant. Hypothesis C was that the 
factor loading pattern across the three subpopulations is 
invariant. Hypothesis D was that the error/unigqueness 
structure is invariant across the three subpopulations. 
Hypothesis E was that the factor variances and covariances 
are invariant across the three subpopulations. 

As mentioned previously in Chapter MIII, the 
chi-square/degrees of freedom ratio is distributed as a 
t-statistic so that anything greater than 1.96 is significant 
and therefore would reject the null hypothesis of invariance. 


Table 4.3 shows the testing results. 


TABLE 4.3 
EQUIVALENCE TESTING 


—SESE]|EES=SSpEnhL—>~>~E=~E SEE Se SS SSS SSS SS ———UeEEEEEeEeEEEEoEEeEeEeEEeEeEEE 
HYPOTHESIS chi 2 dt chi 2/dt LR Test 


A COVARIANCE STRUCTURE tS THE SAME 242 4 182 leas -- 
B NUMBER OF FACTORS IS THE SAME SSN) 3 2 177 1. 87 -- 
C: FACTOR LOADING tS THE SAME 341 0 195 7.5 0.55 
D: THE ERROR/UNIQUENESS tS THE SAME 383 6 221 1.74 1°19 


E FACTOR VARIANCES AND COVAR TANCES ARE THE SAME 394.5 241 1 64 0.99 


6. Post Survey Interviews (Convergent Validity) 
During the one-day visits to the Naval Hospitals Camp 


Lejeune and Jacksonville, time constraints limited the number 


of individuals that were interviewed to a total of 15 
(approximately 5% of the total sample population). eta 
working around individual work schedules, an equal 


distribution of interviews was obtained. Of the total 15 
interviews, three were with physicians, three with 
administrative personnel, three with clinic personnel, three 
with laboratory personnel, and three with pharmacy personnel. 
During the interviews, the individuals were asked about their 
overall impression of satisfaction (satisfied or 
dissatisfied), and to comment about the _ system. All 
individuals interviewed had at least nine-months experience 
with the CHCS. Between interviews, there was an opportunity 
for first-hand use of the CHCS at the Naval Hospital Camp 
Lejeune's training room. 

The majority of the phySicians (2 out of 3) were 
overall dissatisfied with the CHCS. They sited a cumbersome 
menu interface, slow response time of system, and slow 
response time _ to change the system. The clinic, 
administrative, laboratory and pharmacy personnel interviewed 
expressed they were satisfied, but also echoed the same 
comments as the physicians. 

Other general comments offered about the survey 
instrument were: 1) provide an example--the dual bipolar 
adjective pairs tended to confuse some and 2) the instrument 
did not address how the user interacted or put information 


into the system. 
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C. EVALUATION OF THE INSTRUMENT'S DATA 
1. Testing Differences in Means 
The combined three hospital data set was used to test 
the difference in means between the three user groups 
(Physicians, Medical Support, and Administrative Support) for 
each of the 13 questions, overall (summed) score, and factors 
(averaged score). Using the combined data set, each user 


group was tested for differences in means between the three 


location sites (1l.e., Charleston vs Camp Lejeune vs 
Jacksonville). Lastly, each hospital's data was tested for 
differences in means between the three user groups. Figure 


mes Shows the level of satisfaction for each of the 13 
questions and by each user group. 


MEASURE OF USER SATISFACTION 


BY INDIVIDUAL QUESTIONS AND USER GROUPS 












INNO 


LZ. 





ec 
Question!  Q2 Q3 


Physicians G Medical Support Administrative Support 






Figure 4.5. Measure of User Satisfaction 
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a. Combined Hospital Data 
(1) Differences between User Groups 

Table 4.4a shows the ANOVA testing results 
between user groups. Significant findings (at an alpha of 
O05) are underlined. The Physician group sample size 
consisted of 79 participants; the Medical Support group 
consisted of 207 participants; and the Administrative Support 
group consisted of 54 participants. The a posteriori 
testing for normality via _ the NSCORES correlation 
demonstrates that the data has a normal distribution. Except 
for questions 9 and 10, the Hartley Fmax test revealed 
homogeneity of variance between groups. Questions 9 and 10 
were tested using the Kruskal-Wallis nonparametric ANOVA 
testing of sample medians. The nonparametric findings also 
showed no significant difference in medians. Questions 7, 8, 
9, 10, and 13 which make up Factor C (information product 
output) revealed no significant differences. Factor C across 
the three user groups means were within the "slightly 


satisfied" range (0 to 1). 
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TABLE 4.4A 
COMBINED HOSPITALS ANOVA TESTING; USER GROUPS 


























USER GROUP MEANS(1) F VALUE P VALUE coNSCORES Har t ley 

QUESTIONS /FACTORS a —_ = ATION Fmax(2 
p M A 
1. RELATIONSHIP WITH MID STAFF 0.88 eae 1.65 6.22 0.002 0.98 1.12 
2. PROCESSING OF REQUESTS FOR CHANGE -0.85 -0.28 -0.24 4.02 0.019 0.98 1.54 
3. DEGREE OF TRAINING -0.47 0.39 0.12 7.23 0.001 0.99 1.34 
4. _USER“S UNDERSTANDING OF SYSTEM -0.19 0.98 1.03 16.33 0.000 0.98 1.17 
5. USER'S FEELING OF PARTICIPATION -0.11 0.94 0.81 14.24 0.000 0.98 fei 
6. ATTITUDE OF MID STAFF 1.12 1.45 1.79 4.37 0.013 0.98 1.13 
7. RELIABILITY OF OUTPUT 1.10 0.98 0.94 0.29 0.752 0.97 1.41 
8. RELEVANCY OF OUTPUT 0.60 1.04 1.02 2.94 0.054 0.97 114 
9. ACCURACY OF OUTPUT 1.32 1.20 0.81 2.43 0.089 0.97 2.08 
10. PRECISION OF OUTPUT 0.94 0.72 0.60 1.26 0.285 0.98 2.01 
11. COMMUNICATION WITH MID STAFF 0.79 Wale 1.53 4.99 0.007 0.98 1.35 
12. TIME REQUIRED FOR NEW DEVELOPMENT —f) ey 0.05 -0.15 13.05 0.000 0.99 1.26 
13. COMPLETENESS OF OUTPUT 0.45 0.77 0.73 1.28 0.279 0.98 1.16 
OVERALL SCORE 4.54 10.71 10.62 7.31 0.001 0.99 1.69 
A MID STAFF AND SERVICES 0.93 deh 1.66 6.49 0.002 0 98 1.15 
B. CONTRACTOR SERVICES -0.95 =G.12 -0.19 10.45 0.000 0.99 1.35 
C. INFORMATION OUTPUT 0.88 0.94 0.82 0.26 0.772 0.98 1.61 
D. KNOWLEDGE AND INVOLVEMENT -0. 26 0.77 0.65 18.70 0.000 0.99 1.28 
NOTES 


i im. elas m = medical Support: A = Administrative Suppor 
r 


2 tley’s fae 12 approximate critical value = 1.80 at alpha = 0.05 
(2) Significant User Group Findings 
Table 4.4b represents those items where the 


ANOVA testing in Table 4.4a revealed a significant difference 


in the means between the user groups. Scheffe multiple 
comparison testing was used to identify the individual 
differences between user groups. Physicians were less 


satisfied and displayed a significant difference between the 
other two uSer groups in virtually all the individual 
questions that make up Factor A (MID Staff and Services; 
questions 1, 6, and 11), Factor B (Contractor Services; 
questions 2 and 12), and Factor D (User Knowledge and 


Involvement; questions 3, 4, and 5). 


ayl 


TABLE 4.4B 
SCHEFFE MULTIPLE COMPARISON TESTING; USER GROUPS 


Sa —oeoeoEoeEeE—— SSS SSEFYFYFEqS FSS 
USER GROUPS COMPARISON 
QUESTIONS/FACTORS PHYSICIAN/ PHYSICIAN/ MEOICAL SUPPORT/ 
MEDICAL SUPPORT ADMIN. SUPPORT ADMIN SUPPORT 


1. RELATIONSHIP WITH AID STAFF $(1) Ss NS (2) 
2. PROCESSING OF REQUESTS FOR CHANGES Ss NS NS 
3. DEGREE OF TRAINING PROVIDED Ss NS NS 
4. USER’S UNDERSTANDING OF SYSTEM Ss Ss NS 
5. USER'S FEELING OF PARTICIPATION Ss Ss NS 
6. ATTITUDES OF MID STAFF NS Ss NS 
11. COMAUNICATING WITH MID STAFF NS Ss NS 
12. TIME REQUIRED FOR NEW DEVELOPMENT Ss Ss NS 
OVERALL SCORE Ss Ss NS 
A. MID STAFF AND SERVICES Ss Ss NS 
B. CONTRACTOR SERVICES Ss Ss NS 
C. KNOWLEOGE AND INVOLVEMENT Ss Ss NS 


= Significant difference in means 
2 NS = Nonsignicant: no difference in means 


n 
Cc 


(3) Physician Group Differences by Site 

Table 4.5a shows the ANOVA testing results 
for Physician user group between the three Naval hospital 
(NH) sites. Significant findings (at an alpha of 0.05) are 
underlined. The Physician group sample size at the NH 
Charleston consisted of 21 participants; NH Camp Lejeune 
consisted of 22 participants; and NH Jacksonville consisted 
of 36 participants. The a posteriori testing for normality 
via the NSCORES correlation demonstrates that the data has a 
normal distribution. Except for questions 7 and 13, the 
Hartley Fmax test revealed homogeneity of variance between 
groups. Questions 7 and 13 were tested using the 
Kruskal-Wallis nonparametric ANOVA testing of sample medians. 
The nonparametric findings also showed no significant 


difference in medians. 


S2 


Questions 7, 8, 9, 10, and 13 which make up 
Bactor C (information product output) revealed no 
Significant differences. Factor C across the three sites for 
the Physician group were essentially within the "slightly 
satisfied" range (0 to 1). Other nonsignificant differences 
in means were found for questions 4 and 12. Questions 4 and 
12 both were essentially within the "slightly dissatisfied" 
range (-1 to 0), and help make up Factors D and B 


respectively. 
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TABLE 4.54 
PHYSICIAN GROUP ANOVA TESTING 














LOCATION SITE MEANS(1) F VALUE P VALUE NSCORES Hartle 
QUEST IONS/ FACTORS oo CORRELATION Fmax( 2) 
CH CL jx 
1. RELATIONSHIP WITH MID STAFF 1.38 1.36 0.29 9.23 0.000 Oo 98 1.70 
2 PROCESSING OF REQUESTS FOR CHANGE -0 45 70.05 -1.60 8.81 0.000 0.98 1.45 
3. DEGREE OF TRAINING 0.05 0.05 -1.08 5.99 0.004 0.99 1.48 
4 USER'S UNDERSTANDING OF SYSTEM 0.14 -0.59 -0.14 1.26 0.289 0.98 V23 
5. USER'S FEELING OF PARTICIPATION 055 -0.21 -0.44 3.49 0.035 0.97 1.69 
6 ATTITUDE OF MIO STAFF 1.57 Te 0 46 12.06 0.000 0.97 1:72 
7. RELIABILITY OF OUTPUT 1.07 1.32 0.99 0.47 0.625 0.95 3.13 
8 RELEVANCY OF OUTPUT 0.57 0.71 0.54 0.09 0.913 0.96 1.10 
9. ACCURACY OF OUTPUT 1.41 1.50 1.17 0.80 0 451 ORS, 1.98 
10. PRECISION OF OUTPUT 0 86 0.86 1.04 0.32 0.728 0.98 1.76 
11) COMMUNICATION WITH MID STAFF 1.36 1.34 0.13 13.61 Q.000 0.99 1 40 
12. TIME REQUIRED FOR NEW DEVELOPMENT -0 93 -0 52 “1 42 2 61 0 075 0 98 the UE 
13. COMPLETENESS OF OUTPUT 0.52 O 84 0.17 15 Oa227 0 98 S.07 
OVERALL SCORE 8 10 8.39 O.11 Z.2t 0.001 0 97 1 68 
A MID STAFF AND SERVICES 1 44 1 49 0.29 14.69 0.001 0.98 1.24 
B CONTRACTOR SERVICES -0 69 -0.28 -1.50 7.38 0.001 0.99 1 74 
C. INFORMATION OUTPUT 0 86 1.04 0.78 OFS 0 605 0.97 2.14 
O. KNOWLEDGE AND INVOLVEMENT Ve Pas) -0.25 -0.56 3.23 0.045 0.99 2.05 


NOTES - 
1 CH = NH Charleston: CL = NH Camp Lejeune: JX = NH FICASLSES A li 
2) Hartley's Fmax;, 97) approximate critical vatue = 2.57 at alpha = 0 05 


(4) Significant Physician Group Findings 

Table 4.5b represents those items where the 
ANOVA testing in Table 4.5a revealed a significant difference 
in the means between the Physician groups. Scheffe multiple 
comparison testing was used to identify the individual 
differences between sites. 

Physicians at the NH Jacksonville were less 
Satisfied and displayed a significant difference between the 
other two sites in all the individual questions (1, 6, and 
11) that make up Factor A (MID Staff and Services). The NH 
Charleston and the NH Camp Lejeune Factor A findings were 
within the "quite satisfied" range (1 to 2) versus the NH 
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Jacksonville's findings within the "slightly satisfied" range 
fom to 1). Physicians at the NH Jacksonville were less 
satisfied and displayed a significant difference between the 
other two sites for Factor B (Contractor Services; questions 
2 and 12). Physicians at the NH Jacksonville were less 
satisfied as compared with the NH Charleston with Factor D 
(Knowledge and Involvement; questions 3, 4, 5), but 
demonstrated no difference between the NH Camp Lejeune. The 
NH Jacksonville, and the NH Camp Lejeune findings for Factor 
D were within the "slightly dissatisfied" range (-1 to 0) 
verses the NH Charleston's findings within the "slightly 


satisfied" range (0 to 1). 
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TABLE 4.5B 
SCHEFFE MULTIPLE COMPARISON TESTING; PHYSICIAN GROUP 


a arr, SSE ie =e 
LOCATION SITE COMPARISONS 


QUEST IONS/FACTDRS CHARLESTON/ CHARLESTON/ CAMP LE JEUNE/ 
CAMP LE JEUNE JACKSONVILLE JACKS ILLE 


RELATIONSHIP WITH MID STAFF NS(1) $(2) s 
PROCESSING OF REQUESTS FOR CHANGES 
DEGREE DF TRAINING PROVIDED 
USER'S FEELING DF PARTICIPATION 

6. ATTITUDES DF AMID STAFF 

11. COMMUNICATION WITH AID STAFF 
OVERALL SCDRE 


ww Ne 


GaEEGhE SG 


3 
9 
NS 
s 
Ss 
= 


wUNHUNnYH 


4) 


A. MID STAFF AND SERVICES 
B. CONTRACTDR SERVICES 
D. KNOWLEDGE AND INVDLVEMENT 


Ga G 


ay } ENS = Nonsignifican no eae i. means 
S = Significant anineren nce in me 


Ne eeeeeEEe——ee—e—eEE——EE 
(S) Medical Support Group Differences by Site 
Table 4.6a shows the ANOVA testing results 
fe Chis Medical Support user group between the three naval 
hospital (NH) sites. Significant findings (at an alpha of 
0.05) are underlined. The Medical Support group sample size 
at the NH Charleston consisted of 71 participants; at the NH 
Camp Lejeune consisted of 75 participants; and at the NH 
Jacksonville consisted of 61 participants. The a posteriori 
testing for normality via the NSCORES correlation demonstrates 
that the data has a normal distribution. The Hartley Fmax 
test revealed homogeneity of variance between groups. 
Questions 7, 8, 9, 10, and 13 which make up 
Factor C (information product output) revealed no significant 
differences. Factor C across the three user groups means were 
essentially within the "slightly satisfied" range (0 to 1). 


Other nonsignificant differences in means was found for 


56 


question 4. Question 4 was essentially within the "slightly 


dissatisfied" range (0 to -1), and helps make up Factor D. 


TABLE 4.6A 
MEDICAL SUPPORT GROUP ANOVA TESTING 























LOCATION SITE MEANS(1) F VALUE P VALUE NSCORES Hartle 

QUESTIONS/FACTORS eo ee CORRELATION Fmax (2) 
CH cr JX 
1. RELATIONSHIP WITH MID STAFF 1.74 1.22 0.94 6.88 0.001 0.98 1.24 
2. PROCESSING OF REQUESTS FOR CHANGE -0.42 0.24 -0.76 6.79 0.001 0.99 1.69 
3. DEGREE OF TRAINING 0.82 0.54 -0.30 7.37 0.001 0.98 1.05 
4 USER'S UNDERSTANDING OF SYSTEM Usd 0.82 0.77 24 Be! 0.078 0.97 1.42 
5. USER’S FEELING OF PARTICIPATION 1.25 0.99 0.53 3.66 0.027 0.98 1.32 
6. ATTITUDE OF MID STAFF 2.08 Us die) 0.85 16.91 0.000 0.98 1.98 
7. RELIABILITY OF OUTPUT 0.80 | a 2z/ 0.81 745 0.099 0.97 1 48 
8. RELEVANCY OF OUTPUT t.06 1.23 0.76 1.90 0.151 0.98 1.26 
9, ACCURACY OF OUTPUT 1.31 la de 0.91 1.68 0.189 0.97 1.20 
10. PRECISION OF OUTPUT 0.58 0.79 0.79 0.53 0.591 0.97 Weaea 
11. COMMUNICATION WITH MID STAFF ols 1.03 0.68 10.92 0.000 0.98 1.64 
12. TIME REQUIRED FOR NEW DEVELOPMENT 0.10 0.34 ~0.36 3.08 0.048 0.99 1.32 
13. COMPLETENESS OF OUTPUT 0 68 0.94 0.67 0.63 0.536 0.98 1.21 
OVERALL SCORE 13 06 12.08 6.30 5.02 0.007 0.99 1 23 
A. MID STAFF ANO SERVICES 1.85 1.20 0.83 14.10 OQ .000 0.98 1.64 
B. CONTRACTOR SERVICES -0.16 0.29 -0.56 6.09 0.003 0.99 1.47 
C. INFORMATION OUTPUT 0.89 1.11 0.79 1.27 0.284 0 98 1.26 
D. KNOWLEDGE AND INVOLVEMENT ia 0.78 0.33 6.36 0.002 0.99 1.29 
NOTES 


MES): 
ly CH = fila Who ead CL = NH Camp Lejeune; JX = NH jJacksonvi 
$ x 


(6) Significant Medical Support Group Findings 
Table 4.6b represents those items where the 
ANOVA testing in Table 4.6a revealed a significant difference 
in the means between the Medical Support groups. Scheffe 
multiple comparison testing was used to identify the 
individual differences between sites. For Factor A (MID 
Staff and Services) comprised of questions 1, 6, and 11, the 


NH Charleston was significantly different from the other two 
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Sites. In each of the individual questions the NH Charleston 
was more satisfied ("quite satisfied" range of 1 to 2) than 
the two sites which were within the "slightly satisfied" 
range (0 to 1). For Factor B (Contractor Services), which 
was comprised of questions 2 and 12, the NH Jacksonville was 
less satisfied when compared with the NH Camp Lejeune, but had 
no significant difference between the NH Charleston. Both 
the NH Jacksonville and the NH Charleston findings for Factor 
B were within the "slightly dissatisfied" range (0 to -1), 
whereas the NH Camp Lejeune findings for Factor B were within 
the "slightly satisfied" range (0 to 1). Even though the NH 
Charleston and the NH Camp Lejeune have opposing findings for 
Factor B, they were still statistically nonsignificant for 
differences between their respective means. For Factor D 
(User Knowledge and Involvement), the NH Jacksonville scored 
less satisfied findings as compared with the NH Charleston, 
but no significant difference between the NH Camp Lejeune. 
However, all three sites for Factor D_ scored within 


essentially the "slightly satisfied" range (0 to 1). 
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TABLE 4.6B 
SCHEFFE MULTIPLE COMPARISON TESTING; MEDICAL SUPPORT GROUP 


LOCATION SITE COMPARISONS 


QUESTIONS/ FACTORS CHARL ESTON/ CHARLESTON/ CAMP LE JEUNE/ 
CAMP LE JEUNE JACKSONVILLE JACKSONVILLE 


1. RELATIONSHIP WITH MID STAFF $1) NS(2) 
2. PROCESSING OF REQUESTS FOR CHANGES NS 

3. DEGREE OF TRAINING PROVIDED NS 

5. USER'S FEELING OF PARTICIPATION NS 
6. ATTITUDES OF MID STAFF S 
11. COMMUNICATING WITH MID STAFF S 
12. TIME REQUIRED FOR NEW DEVELOPMENT NS 
OVERALL SCORE NS 


“A665 444 


A. MID STAFF AND SERVICES S 

B. CONTRACTOR SERVICES NS NS 
D. KNOWLEDGE AND INVOLVEMENT NS 

NOTES: 

3} S = Signi 
2) NS = NONsi 


AY & 


act difference in mea 


t icar ns 
gniticant: no difference in means 


ee ee ee 
(7) Administrative Group Differences by Site 

Table 4.7a shows the ANOVA testing results 

for Administrative Support user group between the three naval 

hospital (NH) sites. Significant findings (at an alpha of 

0.05) are underlined. The Administrative Support group 


Sample size at the NH Charleston consisted of 9 participants; 


NH Camp Lejeune consisted of 24 participants; and NH 
Jacksonville consisted of 21 participants. The a posteriori 
testing for normality via the NSCORES’- correlation 


demonstrates that the data has a normal distribution. Except 
for Factor B, the Hartley Fmax test revealed homogeneity of 
Variance between groups. 

Factor B which is the averaged score of 


questions 2 and 12, was tested using the Kruskal-Wallis 


nonparametric ANOVA testing oof sample medians. The 
nonparametric findings showed significant difference in 
medians (P value = Oys0 sue However, the Mann-Whitney 


2) 2) 


nonparametric pairwise comparisons were the same as the 
Scheffe parametric findings as shown in Table 4.7b. 

Factor D dealing with training, knowledge and 
involvement with the CHCS system was nonsignificant for 
differences in means between the three sites. Factor D is the 
average score of questions 3, 4, and 5. Factor D score was 


within the "slightly satisfied" range (0 to 1). 


TABLE 4.7A 
ADMINISTRATIVE SUPPORT GROUP ANOVA TESTING 

















LOCATION SITE MEANS(1) F VALUE P VALUE NSCORES Hartle 
QUEST IONS/FACTORS SIE EE EE CORRELATION Fmax(2 
CH CL yx 
1 RELATIONSHIP WITH MID STAFF 1 8&9 2.19 0 93 7.22 0.002 0 97 1 67 
2 PROCESSING DF REQUESTS FDR CHANGE -1 28 0.25 -0.36 5.04 0.010 0 97 {] . Bee 
3. DEGREE DF TRAINING -0.11 0.50 -O 21 1 20 0.308 Oo 99 174 
4 USERS UNDERSTANDING DF SYSTEM 0.00 dis! i) 27% 2 24 O 117 Oo 98 1 41 
5 USER’S FEELING DF PARTICIPATION 0 89 131 0 19 3.35 0,043 0.99 174 
6. ATTITUDE OF MID STAFF Zea 2 42 oO 8&8 13.87 0.000 O 96 1.85 
7. RELIABILITY OF OUTPUT O 06 le 24/ 0 93 2 91 0.064 Oo 98 ) 7: 
8 RELEVANCY DF OUTPUT 1.00 1.35 0.64 1.60 0.212 O 97 1 44 
9 ACCURACY OF OUTPUT -0 50 1 44 0 64 7.85 0.001 0 98 1 49 
10. PRECISION OF OUTPUT -0.44 1 08 Oo 50 $17 0.009 0 98 2.21 
11. COMMUNICATION WITH MID STAFF in 247 071 8.55 0.001 0 97 114 
12) TIME REQUIRED FDR NEW DEVELDPMENT ~1.06 ),, We! -0 07 1.83 0 171 0.99 1 48 
13. COMPLETENESS DF OUTPUT 0.06 1.04 0.67 1 56 0 220 Oo 98 129 
DVERALL SCDRE 4.44 16 48 6 57 5.89 0,005 0 99 1.68 
A. MID STAFF AND SERVICES 195 2 26 0 84 12.09 0.000 0 97 1.24 
B. CONTRACTOR SERVICES -1 17 0 19 -0 21 4.34 0.018 Oo 99 1.74 
C. INFORMATION OUTPUT 0.03 1.24 0 68 4.66 0.014 0.98 1.56 
D. KNOWLEDGE AND INVOLVEMENT O 26 1 05 0.37 2 04 0.141 0 99 1.19 
NOTES: 


val ie reek te As la CL = NH Camp Lejeune: jX = NH Jacksonvil 
r 


le 
2 ey 's cama 17) approximate critical value = 3.30 at alpha = 0.05 


(8) Significant Administrative Support Group 
Findings 


Table 4.7b represents those items where the 


ANOVA testing in Table 4.7a revealed a significant difference 
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in the means between the Administrative Support groups. 
Scheffe multiple comparison testing was used to identify the 
individual differences between sites. 

The Administrative Support group at the NH 
Jacksonville for Factor A (MID Staff and Services; questions 
1, 6, and 11) were less satisfied than the other two sites. 
The NH Camp Lejeune and the NH Charleston were both 
essentially within the "quite satisfied" range (1 to 2), 
whereas the NH Jacksonville was within the "slightly 
satisfied" range (0 to 1). 

The NH Charleston for Factor B (Contractor 
Services; questions 2 and 12) was less satisfied than the 
other two sites. The NH Camp Lejeune for Factor B was within 
the "slightly satisfied" range (0 to 1), and the NH 
Jacksonville was within the "slightly dissatisfied" range (0 
to -1), but was not statistically different. The NH 
Charleston for Factor B was within the "quite dissatisfied" 
mange (—-l1 to -2). 

BOGmMbaceOre Cer nfOormation Product Output; 
questions 7, 8, 9, 10, and 13), the NH Charleston was less 
satisfied as compared with the NH Camp Lejeune, however, 
scored no difference with the NH Jacksonville. The 
Significant influence to this finding was the significant 
differences paralleled with questions 9 and 10 dealing with 
output accuracy and precision. Both the NH Camp Lejeune and 


the NH Jacksonville were essentially within the "slightly 


61 


Satisfied" range (0 to 1) for Factor C, whereas, the NH 


Charleston was "neutral" with a zero score. 


TABLE 4.7B 
SCHEFFE MULTIPLE COMPARISON TESTING; ADMIN SUPPORT GROUP 


SS SS Se——=“———ss—sSsSsSs ee La 
LOCATION SITE COMPARISONS 


QUEST !ONS/ FACTORS CHARLES TON/ CHARLESTON/ CAMP LE JEUNE/ 
CAMP LE JEUNE JACKSONVILLE JACKSONVILLE 


RELATIONSHIP WITH MID STAFF NS( 1} NS $(2) 
PROCESSING OF REQUESTS FOR CHANGES S NS 
USER'S FEELING OF PARTICIPATION NS NS 
ATTITUDES OF MID STAFF NS S 
ACCURACY OF OUTPUT S NS 
10. PRECISION OF OUTPUT S NS 
11. COMMUNICATION WITH MID STAFF NS NS 
OVERALL SCORE S NS 


uo oA WwW WN = 


nun gun & 


A MID STAFF AND SERVICES NS S S 
B. CONTRACTOR SERVICES S S NS 
C. INFORMATION OQUFPUT S NS NS 


NOTES: 
a NS = peri IMME E no difference in means 
(2) S = Significant difference in means 


b. The Naval Hospital Charleston 
(1) Differences between User Groups 

Table 4.8a shows the ANOVA testing results 
between user groups. Significant findings (at an alpha of 
OGS)) are underlined. The Physician group sample size 
consisted of 21 participants; the Medical Support group 
consisted of 71 participants; and the Administrative Support 
group consisted of 9 participants. The a posteriori testing 
for normality via the NSCORES correlation demonstrates that 
the data has a normal distribution. The disparity in the 
size of the user groups sample sizes lead to reject 


homogeneity of variance in a number of questions. However, 
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on subsequent testing using nonparametric procedures revealed 


essentially the same results and patterns. 


TABLE 4.8A 
NH CHARLESTON USER GROUP ANOVA TESTING 








USER GROUP MEANS(1) F VALUE P VALUE NSCORES Hartle 
QUESTIONS /FACTORS ST CORRELATION Fmax( 25 
Pp iy A 
1. RELATIONSHIP WITH MIO STAFF 1 38 1.74 1 89 O 84 0 433 oO 98 1 38 
2. PROCESSING OF REQUESTS FOR CHANGE -0 45 -0 42 -1.28 1.08 0.345 0 99 2.48 
3. DEGREE OF TRAINING 0.05 0.82 -O.11 2 49 0.088 0 98 1.98 
4 USER'S UNDERSTANDING OF SYSTEM 0.14 1.33 0.00 6.99 o.001 Oo 98 1 39 
5. USER S FEELING OF PARTICIPATION 0.55 1.25 0.89 1.70 O 188 0 97 1.46 
6. ATTITUOE OF MID STAFF 1). Be/ 2 08 7) 7405 2 41 0.095 Oo 98 1 31 
7. RELIABILITY OF OUTPUT 1.07 Oo 80 0.06 53 0.221 Oo 98 3.99 
& RELEVANCY OF OUTPUT 0.57 1.06 1.00 1.02 0.366 0 98 1.39 
9. ACCURACY OF OUTPUT 1 41 1.31 -0.50 6.82 0.002 0 98 2.59 
10. PRECISION OF OUTPUT 0. 86 0.58 -O 44 2a, 0.068 Oo 98 3.81 
11 COMMUNICATION WITH MID STAFF 1 36 iv ee: 0.89 O 413 Oo 98 1 18 
12. TIME REQUIREO FOR NEW OFVELOPMENT -0 93 0.10 -1.06 4.79 0.010 Oo 99 1.37 
13. COMPLETENESS OF OUTPUT O 52 O 68 0.06 0.78 0 459 0 99 3.98 
OVERALL SCORE 8 10 13 06 4 44 3.22 0.044 0.99 2.19 
A. MIO STAFF AND SERVICES 1 44 1 85 1 95 1 58 O 211 0 98 We 
B. CONTRACTOR SERVICES -0 69 -0.16 -1 17 3.19 0,045 0 99 3.76 
C INFORMATION OUTPUT Oo 89 Oo 89 0.03 223 OFS 0 98 1 85 
O. KNOWLEDGE AND INVOLVEMENT 0.25 eee 0.26 5.79 0.004 0 99 1) 22 
NOTES 
P Physicians: M = Medical Support, A = administrative Suppor 


) a 


1 = v t 
>} Hartley s Fmax;4 43, approximate critical value = 2 35 at alpha = 0.05 


(2) Significant User Group Findings 
Table 4.8b represents those items where the 


ANOVA testing in Table 4.8a revealed a significant difference 


in the means between the user groups. Scheffe multiple 
comparison testing was used to identify the individual 
differences between user groups. Physicians were less 


Satisfied when compared with the Medical Support group for 
Factor D (User Knowledge and Involvement); no difference with 
the Administrative Support group. ° Physicians and the 


63 


Administrative Support group were within the "slightly 
satisfied" range (0 to 1). Whereas, the Medical Support 
group was within the "quite satisfied" range (1 to 2). Factor 
B dealing with Contractor services (questions 2 and 12), on 
multiple comparison testing found that there was no 
Significant difference between the groups. All groups for 
Factor B were essentially within the "Slightly dissatisfied" 
range (0 to -1). Again, the small sample size of the 
Administrative Support group plays a role in the resulting 
nonsignificant findings. 

Interesting, the Administrative Support group 
were less satisfied than the other two groups as to the 
accuracy of the output (question 9). However, there were 
nonsignificant differences between all groups found in Factor 


C which is made up of questions 7, 8, 9, 10, and 13. 


TABLE 4.8B 
SCHEFFE MULTIPLE COMPARISON TESTING; NH CHARLESTON 
USER GROUPS 


ee EEE 
USER GROUPS COMPARISON 


QUESTIONS / FACTORS PHYS ICLAN/ PHYS ICLAN/ aco SUPPORT/ 
MEOTCAL SUPPORT ADMIN SUPPORT ADMIN SUPPORT 


4. USER’S UNDERSTANDING OF SYSTEM $(1) NS(2) NS 
9. ACCURACY OF OUTPUT NS s S 
12. TIME REQUIREO FOR NEW OEVELOPMENT Ss NS NS 
OVERALL SCORE NS NS NS 


B. CONTRACTOR SERVICES NS NS 
O. KNOWLEDGE AND INVOLVEMENT Ss NS 


G&G 


= Significant dl fference In me 
= Nonsignificant: no aLeperence= in means 
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c. Naval Hospital Camp Lejeune 
(1) Differences between User Groups 

Table 4.9a shows the ANOVA testing results 
between user groups. Significant findings (at an alpha of 
0.05) are underlined. The Physician group sample size 
consisted of 22 participants; the Medical Support group 
consisted of 75 participants; and the Administrative Support 
group consisted of 24 participants. The a posteriori testing 
for normality via the NSCORES correlation demonstrates that 
the data has a normal distribution. Except for questions 6 
and 9, the Hartley Fmax test revealed homogeneity of variance 
between groups. However, on subsequent testing using 
nonparametric procedures, resulted in finding essentially the 
same values and patterns. 

Although, the Physician group's mean value for 
Factor B (Contractor Services; questions 2 and 12) was within 
the "slightly dissatisfied" range (0 to -1), and the other 
two groups were within the "slightly satisfied" range of (0 
to 1), there was no significant difference in means between 
the groups. There was no significant difference between the 
three groups concerning the information product output 
(Factor C; questions 7, 8, 9, 10, and 13). All three groups 
were within the "quite satisfied" range (1 to 2) for Factor 


. 
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TABLE 4.9A 
NH CAMP LEJEUNE USER GROUP ANOVA TESTING 

















USER GROUP MEANS(1} F VALUE P VALUE NSCORES Hartle 
QUESTIONS/FACTORS a CORRELATION Fmax(2) 
P M A 
1. RELATIONSHIP WITH AIO STAFF 1.36 1.22 2.19 5.78 GQ. 004 0.98 1.76 
2. PROCESSING OF REQUESTS FOR CHANGE -0 05 O 24 0.25 0.37 0.690 0.98 (ig 735) 
3. DEGREE OF TRAINING 0.05 0.52 0.50 0.75 0.475 0.99 1.07 
4. USER'S UNDERSTANDING OF SYSTEM -0 59 0 82 1.33 7.95 0.001 0 98 We 
5. USER’S FEELING OF PARTICIPATION -0.21 0.99 1.31 7.98 0.001 0.99 1.47 
6 ATTITUDE OF MID STAFF 1.77 1.33 2.42 7.09 0.001 0.98 2.32 
7. RELIABILITY OF OUTPUT Us 874 {2 737/ 1 22z/ 0.01 0.989 0.96 1.31 
8. RELEVANCY OF OUTPUT 0.71 1.23 1.35 1.64 0.199 0.98 1.30 
9 ACCURACY OF OUTPUT 1.50 1.33 1.44 0.19 0.828 0.98 2.85 
10. PRECESION OF OUTPUT 0 86 0.79 1.08 0.45 0.637 0 97 1.54 
11. COMMUNICATION WITH MIO STAFF 1.34 1.03 2417 6.59 0.002 0 98 2.07 
12 TIME REQUIREO FOR NEW DEVELOPMENT -0 52 0.34 0 13 2.63 0.076 0 99 1,50 
13. COMPLETENESS OF OUTPUT O 84 0 93 1.04 0.11 0 897 0 98 1.41 
OVERALL SCORE 8 39 12 08 16 48 2-53 O 084 0 99 1.54 
A MIO STAFF AND SERVICES 1 49 1.20 2.26 &.45 0.000 0.98 1.43 
B. CONTRACTOR SERVICES -0.28 0.29 0.19 1 64 0.199 0.99 Tet 
C. INFORMATION OUTPUT 1 O05 1.443 1.24 0 20 0.823 0.99 1 39 
O. KNOWLEDGE AND INVOLVEMENT -0 25 0.78 1.05 6.61 0.002 0.99 121 
NOTES . 
sari )eaeicdertestra suppor tilglaieve ean aria 
(3 39) 


(2) Significant User Group Findings 

Table 4.9b represents those items where the 
ANOVA testing in Table 4.9a revealed a significant difference 
in the means between the user groups. Scheffe multiple 
comparison testing was used to identify the individual 
differences between user groups. 

The Medical Support group was less satisfied 
when compared with the Administrative Support group for Factor 
A (MID Staff and Services; questions 1, 6, 11), and no 
Significant difference with the Physician group. The Medical 
Support group and the Physician group for Factor A were 
within the "quite satisfied" range (1 to 2), whereas the 
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Administrative Support group was within the "extremely 
satisfied" range (2 to 3). The Physician group was less 
satisfied for Factor D (User Knowledge and Involvement; 
questions 3, 4, 5) than the two other groups. The Physician 
group for Factor D were within the "slightly dissatisfied" 
range (0 to -1), whereas, the other two groups’ were 


essentially within the "slightly satisfied" range (0 to 1). 


TABLE 4.9B 
SCHEFFE MULTIPLE COMPARISON TESTING; NH CAMP LEJEUNE 
USER GROUPS 


ee ee KT ee 
USER GROUPS COMPARISON 


QUEST IONS/F ACTORS PHYSICIAN/ PHYS ICIAN/ MEDICAL SUPPORT/ ADMIN 
MEDICAL SUPPORT ADMIN SUPPORT SUPPORT 


RELATIONSHIP WITH MID STAFF NS(1) NS $(2) 
USERS UNDERSTANDING OF SYSTEM S S NS 
USER S FEELING OF PARTICIPATION Ss S NS 
ATTITUDES OF MID STAFF NS NS S 
1. COMMUNICATION WITH MID STAFF NS NS Ss 


- OO wh =~ 


A. MIS STAFF AND SERVICES S NS S 
D. KNOWLEDGE AND INVOLVEMENT S S NS 


NOTES. Ppemeatice’ ' 
Sy NS = Nonsignificant: no difference in means 
2) § = Significant difference in means 

OD C8 OO STZ 


d. Naval Hospital Jacksonville 
(1) Differences between User Groups 

Table 4.10a shows the ANOVA testing results 
between user groups. Significant findings (at an alpha of 
O05) are underlined. The Physician group sample size 
consisted of 36 participants; the Medical Support group 
consisted of 61 participants; and the Administrative Support 
group consisted of 21 participants. The a posteriori testing 


for normality via the NSCORES correlation demonstrates that 
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the data has a normal distribution. The Hartley Fmax test 
revealed homogeneity of variance between user groups. 

There was no Significant difference between 
user group means for Factor C (Information Product Output; 
questions 7, 8, 9, 10, and 13). All user groups were within 
the "Slightly satisfied" range (0 to 1) for Factor C. 
Question 3 dealing with the degree of training was found 
Hiitelaldv stoma vc Significant differences in user group 
means. However, on subsequent testing as denoted in Table 
4.10b, there was no significant difference. All user groups 
for question 3 were essentially within the "slightly 


dissatisfied" range (0 to -1). 
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TABLE 4.10A 
NH JACKSONVILLE USER GROUP ANOVA TESTING 




















USER GROUP MEANS(1) F VALUE P VALUE NSCORES Hartie 
QUESTIONS/FACTORS a aS CORRELATION Fmax(2 
P MA A 
1. RELATIONSHIP WITH MIO STAFF 0.29 0.94 0.93 3.70 0.028 0.99 1,44 
2. PROCESSING OF REQUESTS FOR CHANGE -1.58 -0.76 -0.36 5.18 0.007 0.99 2.19 
3. OEGREE OF TRAINING -1.08 -0.30 -0.21 3.279 0.041 0.99 1.58 
4. USER'S UNDERSTANDING OF SYSTEM -0.14 0.77 1.12 2.80 0.004 0.98 1.48 
5. USER'S FEELING OF PARTICIPATION -0 44 0.53 0.19 4.81 o.oo 0.99 2.00 
6. ATTITUDE OF MIO STAFF 0.46 0.85 0.88 1.34 0.265 0.98 1.89 
7. RELIABILITY OF OUTPUT 0.99 0.81 0.93 0.17 0.846 0.97 1.12 
8. RELEVANCY OF OUTPUT 0.54 0.76 0.64 0.26 0.772 0.98 1.07 
9. ACCURACY OF OUTPUT Us U2 0.91 0.64 1.02 0.364 0.97 154 
10. PRECISION OF OUTPUT 1.04 0.79 0.50 1.34 0.266 0.99 1.81 
11. COMMUNICATION WITH MID STAFF 0.13 0.68 0.71 2.61 0.078 0.99 2.01 
12. TIME REQUIRED FOR NEW OEVELOPMENT -1.42 -0.36 -0 .07 6.08 0.003 0.99 1.40 
13. COMPLETENESS OF OUTPUT 0.17 0.67 0.67 1.15 0.319 0.97 +.64 
OVERALL SCORE 0.11 6.30 6.57 3.19 0.045 0.99 2.22 
A. MIO STAFF AND SERVICES 0.29 0.83 0.84 2.96 0.056 0.98 1,74 
B. CONTRACTOR SERVICES -1.50 -0.56 -0.21 reer té 0.001 0.99 2.09 
C. INFORMATION OUTPUT 0.78 0.79 0.68 0.07 0.931 0.98 1.39 
D. KNOWLEDGE AND INVOLVEMENT -0.56 0.33 0.37 6.50 0.002 0.99 2.09 
(1) = Physicians: MA = Medical Support; A = Administrative Support 
2) Hartley’ 


Vous Ema 33) approximate critical vaiue = 2.25 at alpha = 0.05 
(2) Significant User Group Findings 


Table 4.10b represents those items where the 
ANOVA testing in Table 4.10a revealed a significant difference 
in the means between the user groups. Scheffe multiple 
comparison testing was used to identify the individual 
differences between user groups. 

The Physician group was significantly less 
satisfied for Factor B (Contractor Services; questions 2 and 
12) than the other two user groups. The Medical Support 
group and the Administrative Support group for Factor B were 
within the "slightly dissatisfied" range (0 to -1) whereas, 


the Physician group was within the "quite dissatisfied" range 


69 


(-1 to -2). The Physician group was significantly less 
satisfied for Factor D (User Knowledge and Involvement; 
questions 3, 4, and 5) than the two other user groups. The 
Medical Support group and the Administrative Support group for 
Factor D were within the "slightly satisfied" range (0 to 1) 
whereas, the Physician group was within the "slightly 


dissatisfied" range (0 to -1). 


TABLE 4.10B 
SCHEFFE MULTIPLE COMPARISON TESTING; NH JACKSONVILLE 
USER GROUPS 


USER GROUPS COMPARISON 


QUESTIONS/FACTORS PHYS ICL AN/ PHYSICIAN/ MEDICAL SUPPORT/ 
MEDICAL SUPPORT ADMIN SUPPORT ADMIN SUPPORT 


1. RELATIONSHIP WITH MID STAFF S(1) NS(2) 
2 PROCESSING OF REQUESTS FOR CHANGES S 
3. DEGREE OF TRAINING PROVIDED NS 
4 USERS UNDERSTANDING OF SYSTEM S 
5. USER'S FEELING OF PARTICIPATION = 
12. TIME REQUIRED FOR NEW DEVELOPMENT S 
OVERALL SCORE S 


nu &u gu 
AG G6 G&G GG 


w 


B. CONTRACTOR SERVICES S 
D. KNOWLEDGE AND INVOLVEMENT S S 


& & 


AR in mea 
nt: no dif (erences in means 


2. Time of Use Correlation 
Table 4.11 shows the breakdown of the three hospital 
data for correlation as well as trend analysis mid-points of 
user's level of satisfaction with time of use of the CHCS. 
User time of use of the CHCS was compared against each of 
the four factors (MID Staff and Services, Contractor 
Services, Information Product Output, and User Knowledge and 


Involvement). Significant findings are underlined. 
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TABLE 4.11 
TIME OF USE CORRELATION AND TRENDS 
(COMBINED THREE HOSPITAL DATA) 


CORRELATION TRENDS BY © MONTH INTERVALS 
PEARSON'S 
FACTOR/GROUP CORRELATION aera 0-5 6-11 12-17 184 
COEFFICIENT (6) 
FACTOR A: 

ALL GROUPS -0.016 0.38 1.29 1.31 1.39 1.14 
SAMPLE SIZE 340 -- 67 98 81 94 
PERCENT OF -- -- 20% 29% 23% 28% 

PHYSICIANS -0.080 0.24 0.94 513 0.81 0.76 
SAMPLE SIZE 79 -- 18 25 19 7 
PERCENT OF -- -- 23% 32% 24% 21% 

MEDICAL SUPPORT -0.084 0.11 1.45 1227 1.55 1.03 
SAMPLE SIZE 207 -- 39 60 52 56 
PERCENT OF -- -- 19% 29% 25% 27% 

ADMIN SUPPORT 40.133 0.16 1.30 1.82 1.62 1.74 
SAMPLE SIZE 54 -- 10 13 10 21 
PERCENT OF “+ -- 19% 24% 19% 38% 

FACTOR B: 

ALL GROUPS -0.041 +0.22 -0.18 =0.23 -0.38 -0.47 

PHYSICIANS -0.084 +0. 23 -0.83 -0.88 -0.80 1.32 

MEDICAL SUPPORT -0.105 +0.06 40.24 -0.02 -0.29 -0.29 

ADMIN SUPPORT +0.122 +0.18 -0.63 +0.04 0.00 -0.23 

FACTOR C: 

ALL GROUPS +0 .027 +0.30 40.77 41.03 +0.92 +0.87 

PHYSICIANS 40.057 +0.30 40.58 +1.02 41.13 +0.74 

MEDICAL SUPPORT -0.045 +0.25 40.24 -0.02 -0.29 -0.29 

ADMIN SUPPORT +0. 268 +0.03 -0.08 41.19 +0.74 41.05 

FACTOR D: 

ALL GROUPS +0.030 +0.29 40.41 +0 64 +0.53 40.44 

PHYSICIANS -0.125 +0. 13 +0.01 -0.23 -0.39 -0.43 

MEDICAL SUPPORT 40.014 40.42 +0.67 40.89 40.90 +0.60 

ADAIN SUPPORT +0.120 #0.19 40.12 41.17 +0 .32 40.75 


The Administrative Support group demonstrated a 
significant positive correlation (Pearson's sample correlation 
coefficient r=.27, P value=0.03) for time of use of the CHCS 
and Factor C (Information Product Output). These findings 
suggest that as the time of use increases, the Administrative 
Support group's level of satisfaction with the information 
product output increases. There were no other significant 
correlations noted. 

Table 4.11 shows the breakdown of the three hospital 
data for trend analysis of user's level of satisfaction at 
six month interval time periods of use of the CHCS. A word 


of caution about trend analysis. The trend analysis points 
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are derived from the mean of the respective scores in each 
six month time series interval. One form of time series 
trend analysis attempts to fit a straight line (linear 
relationship trend line) between these points, and is 
generally used in forecasting. As such, these findings are 
only suggestive and based on observable straight line fitting 
of the data. The six month intervals used to look at trends 
are fairly evenly distributed for each of the groups so as to 
limit a weighting problem. 
a. Factor A (MID Staff and Services) Trends 

Figure 4.6 shows the plotting of the six month 
mean scores by All Groups combined as well as each of the 
individual user groups for Factor A (MID Staff and Services) 
trends. Although all scores demonstrate at least within the 
"slightly satisfied" range (0 to 1), the Physician and 
Medical Support user groups appear to have a downward trend. 
This suggests that as Physician and Medical Support personnel 
increase the time of use of the system, their level of 
satisfaction with the local MID staff and services 
diminishes; whereas, the Administrative Support user group 
appears to have an upward trend. This suggests that as 
Administrative Support personnel increase the time of use of 
the system, their level of satisfaction with the local MID 


staff and services increases. 
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percentage of users in this study, and this contributes to 
the very noticeable downward trend for the All Group 


category. 


TREND ANALYSIS 


Factor B (Contractor Services) Satisfaction Scores 


a 
YZ 


All Groups Physicians Medical Support Admin Support 
Less Than 6 Months WY; 6 to 11 Months 12 to 17 Months WY), 18+ Months 


Figure 4.7. Trend Analysis Factor B 





c¢. Factor C (Information Product Output) Trends 

Figure 4.8 shows the plotting of the six-month 
mean scores by All Groups combined as well as each of the 
individual user groups for Factor C (Information Product 
Output) trends. The Medical Support user group appears to 
have a downward trend. This suggests that as Medical Support 
personnel increase the time of use of the system, their 
level of satisfaction with the information product output 
diminishes. As expected, the Administrative Support user 


group appears to have a sharp upward trend (this factor also 
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participants in the system's development. The Physician user 
group appears to have a downward trend. This suggests that 
as physicians increase the time of use of the system, their 
level of satisfaction with understanding and being able to 
use the system diminishes. The Administrative Support user 
group appears to have an upward trend. This suggests that as 
administrative support personnel increase the time of use of 
the system, their level of satisfaction with understanding 


and being able to use the system increases. 


TREND ANALYSIS 


Factor D (Knowledge and Involvement) Satisfaction Scores 
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Figure 4.9. Trend Analysis Factor D 
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V. ANALYSIS OF FINDINGS 


A. INTRODUCTION 

It became apparent while conducting this study that the 
Baroudi and Orlikowski [2] short-form UIS instrument was 
inadequate to assess the overall user satisfaction level for 
the CHCS in this setting. As a result, the significance and 
reporting of the overall UIS scores have been purposely 
diminished. Rather, the reporting and analysis of the 
findings is more appropriately contained within the four 
factors found to summarize the 13 questions asked of the 
participants. As with any exploratory study, this small 
sampling is not intended to provide definitive results. 
Instead, the findings here are intended to provide insight 
and basis for further investigation. In discussing the 
findings, the intention will be to deal with more global 
issues. 

In the first section, the homogeneity of the sample, 
responses, and structure of the short-form UIS instrument is 
discussed. This is important as stated earlier in this paper 
for it forms the foundation to be able to make the statements 
regarding the stability of the four factor structure, and in 
discussing differences without the influence of instrument 
bias. Inthe second section, the limitations confronting the 


current short-form UIS instrument and its impact on the 
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overall assessment of UIS is discussed. Thirdly, the survey 
findings of the three hospital data sampling will be 
discussed for distinctive aifferences and similarities 
between the three user groups. Lastly, the effect of length 
of time of use of the system on user satisfaction levels for 


the four factors will be discussed. 


B. HOMOGENEITY OF SAMPLE AND RESPONSES 

As expected, the sample data revealed a relatively 
homogenous group. The ANOVA testing essentially supports the 
homogeneity of the sample. Although there existed a 
Significant difference in the median education level at 
Jacksonville ("bachelor degree" vs "some college" at the 
other two sites) on subsequent testing for correlation of 


education and the level of satisfaction revealed no 


Significant correlation. Igbaria and Nachman [5] in their 
study found similar findings in that education was not 
correlated with user satisfaction. The same can be said 


essentially for the time of use of the system which will be 
amplified on later. 

The homogeneity is further supported when equivalence 
testing for covariance and factor structure of the short-form 
UIS instrument responses was conducted. The finding of 
invariance between the three sites (subpopulations) for: 1) 
covariance, 2) four-factor structure, 3) factor loadings and 
pattern, 4) error/uniqueness, and 5) factor variance and 
covariances, demonstrates the stability of the new structure, 
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eliminates concerns about construct bias, and enhances the 


argument for homogeneity of the sample. 


C. LIMITATIONS WITH THE CURRENT SHORT-FORM UIS INSTRUMENT 
1. Factor structure 

The fundamental change of the current short-form UIS 
instrument from a three-factor construct to a four-factor 
construct seriously jeopardized its reliability and validity. 
The four-factor solution was tested and confirmed to be the 
stable form in this specific case. Interesting, the Ives et 
ee | 3), 22 question survey construct factored into 
essentially the same four-factor identifiers (MID Staff and 
Services, Contractor Services, Information Product Output, 
and User Knowledge and Involvement) as found in this study. 
Ives et al.[3], eliminated the "vendor support" factor due to 
it only having one item loading into it. 

Back when the Ives et al.[3]} UIS instrument construct 
was developed, and later reduced to the current 13 question 
construct by Baroudi and Orlikowski [2], less emphasis was 
given to systems contracted Aes to external vendors. In the 
Department of Defense, more and more large-scale information 
systems development is being contracted out, as is the case 
with the CHCS. 

With the four-factor construct there exists an 
internal reliability problem. Factor B (Contractor Services) 
suffers much the same problem encountered with the Ives et 
al.[3] study, in that too few items make up the factor. The 


Ves) 


Cronbach alpha for Factor B was 0.68 which is below the 
minimum internal reliability of 0:70 that Nunnally [27] 
recommends. The rule of thumb for the number of items (or 
variables) per factor is that there must be at least three 
distinct variables [(23}[27]. Adding the question concerning 
vendor support from the Ives et al. [3] study would give 
Factor B the minimum three variables. 
2. Weighting of Factors 

The overall UIS score for the current short-form UIS 
instrument is determined by the sum of the 13 question's 
averaged scores. This is the correct procedure, however, 
unless each factor has the same number of questions (or 
variables) the overall UIS score is biased to that factor(s) 
with the most variables. The current short-form UIS 
instrument is heavily influenced by Factor Cc (Information 
Product Output) which has five questions making up this 
factor. Looking at the questions that make up Factor C, 
there exists redundancy. Questions 9 (Accuracy) and 10 
(Precision) could be well served by question 7 (Reliability). 
A definition of question 7 could be offered to mean that the 
information product output is reliable, timely, accurate, and 
precise. The same holds for question 8 (Relevancy) which 
could be well served by question 13 (Completeness). This 


would reduce the emphasis on product output to two questions. 
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3. Information Product Output Factor 

As Factor C's name implies, Information Product Output 
only deals with the system's outputs. An important assessment 
in dealing with user's perception of a system needs to 
address the user interface. If the system has a difficult 
user interface it is more likely that the user will 
understand the system less, and require more training in 
order for the user to become accustomed to the machine. The 
term “user-friendly" is used to denote a user interface that 
is logically designed with the user's needs, intuitively 
obvious, flexible, and offers on-line help. After talking 
with some of the users and seeing the system in operation, 
the user interface iS an area of concern in this case. The 
user interface is an area not addressed by the current 
short-form UIS instrument. Feedback from users noted this 
omission. Interesting, the Ives et al.[{3] survey had user 
interface (convenience of access) which had very high 
reliability correlations and factor loaded into their 
four-factor solution. Combining the user interface question 
along with the two output directed questions from above, 
would give Factor C three variables and equality with the 
other factors. 

It should be noted that Factor D may be considered an 
indirect measure of "user-friendliness." Factor D deals with 


user's assessment of training, understanding, and involvement 
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with the system development. This issue will be further 


Giscussed in the next section. 


D. SIGNIFICANT FACTOR DIFFERENCES BY USER GROUPS 
Figure 5.1 shows the measure of user satisfaction by each 
of the four factors and the three user groups. 


MEASURE OF USER SATISFACTION 


BY FACTORS AND USER GROUPS 


MID Staff & Services Contractor Services Information Output Knowledge & Involvement 


Physicians Wy Medical Support Administrative Support 


Figure 5.1. Measure of User Satisfaction 





1. Factor A (MID Staff and Services) 

Each of the user groups was satisfied with the local 
MID staff and services. This factor assesses the user's 
perception toward their relationship and the ability to 
communicate with the Management Information Department (MID). 


Although each of the user groups knows that the system is 
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being designed and implemented by an external vendor, all 
problems, change requests, and training is first handled 
through the local MID. The level of satisfaction tends to go 
up as the level of system interaction complexity goes down. 
That is to say, physicians were significantly less satisfied 
than the other two groups because their system interaction is 
the greatest and hence the greatest interaction with the 
local MID. Next is the Medical Support group, followed by 
the Administrative Support group with the least system 
routines to interact with. Figure 5.1 shows this stepping 
stone cascade effect. 
2. Factor B (Contractor Services) 

Each of the user groups was dissatisfied with the 
contractor's services. This factor assesses the user's 
perception toward the timeliness of incorporating changes to 
the system. The bureaucratic process for change in government 
contractual agreements can be lengthy. Bailey and Pearson 
[18] noted that the variables in this factor were ones that 
were the most frequently cited as causing dissatisfaction. 
With apparently all three user groups wanting the system to 
change, it brings to question the adequacy of the initial 
system's analysis and requirements process. To design a 
system "on the fly" through prototype-like software version 
updates, can lead to cost overruns and delays in delivery. 

As mentioned above, Factor B's reliability alpha is 


below the minimum level desired for exploratory research. 
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Some caution is warranted. However, as can be seen in Figure 
4.5 and questions 2 and 12 that make up Factor B, the results 
for Factor B are a good reflection of the user groups 
perceptions (no equally counter-balancing responses). 

3. Factor C (Information Product Output) 

Each of the user groups was satisfied with the 
information that the system produced. In fact, there is no 
Significant difference between the three user groups as to 
their level of satisfaction as depicted in Figure 5.1. This 
factor asked the user's assessment of the system's 
information output for reliability, relevancy, accuracy, 
precision, and completeness. 

4. Factor D (User Knowledge and Involvement) 


This factor assesses the user's perception of the 


training received, their understanding of the system 
regarding the ability to use it, and their feeling of 
participation with the system's development. Physicians were 


Significantly less satisfied than the two other groups. The 
Physicians were dissatisfied versus the Medical Support and 
Administrative Support groups reporting being satisfied. The 
system is designed to be driven by the Physician. As such, 
the Physician is given the widest access to the system's 


capabilities, and has the most incentive for wanting a user- 


friendly system. Unfortunately, the myriad of menus with 
limited on-line help makes the system in its current 
configuration difficult to use. The short-form UIS 
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instrument used in this’ study as already mentioned, does not 
directly assess the user interface. More training would only 
be a short-term resolve. Recurring training is difficult to 
schedule, and would require additional resources, adding to 
the cost of overhead maintenance for this system (i.e., for 
trainers, space for training, and time away from health care- 
related productivity). Strassmann [44] contends that sending 
users to classes, especially if this done entirely on the 
organization's time and at the organization's training 
center, 1s about the most expensive, least effective way of 
promoting the desired learning. He supports that the system 


should fit people, not the other way around [44]. 


E. TIME OF USE CORRELATION 

The assumption held is that as the length of time of use 
of the system increased, the user's level of satisfaction 
would increase. This assumption is made in the belief that 
the user would realize with time and experience the benefits 
and efficiencies that the system should impart. The only 
Significant correlation was for the Administrative Support 
group with Factor C (Information Product Output). The 
Administrative Support group exhibited a positive correlation 
with the system's output. There were no other significant 
correlations noted. This lack of correlation for the most 
part may be significant. It does say that a new user's level 
of satisfaction for that respective factor is essentially 
(statistically speaking) the same as a long-time user. 
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VI. CONCLUSION 


The Composite Health Care System (CHCS) is a billion 
dollar military hospital information system venture currently 
being undertaken by the Department of Defense. The design 
and implementation of the CHCS is under contract to an 
external vendor. Good management practices deem assessment 
of effectiveness of any implementation effort. One of the 
most commonly used approaches to assess effectiveness of a 
computer-based information system is the surrogate measure of 
user's attitudes. 

In this study, a commonly used, and previously validated 
13 question short-form user information satisfaction (UIS) 
instrument developed by Baroudi and Orlikowski [2], was used 
to assess its ability to assess overall UIS for the CHCS. 
Baroudi and Orlikowski [2, p. 55] make it clear "that the 
short-form measure iS not a universally applicable and 
immutable measure. It thus may be appropriate in various 
Situations to modify the measure to more adequately reflect 
the requirements of the specific organization." 

One of the important findings in this study is that the 13 
question short-form UIS instrument is inadequate to assess the 
overall satisfaction of the CHCS. However, the information 
gained from its use, combined with further investigation can 


assist in the evaluation of effectiveness of the CHCS. The 
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use of a short-form UIS instrument can be a useful tool. 
When properly designed, it offers the information systems 
manager a quick and easy tool to assess areas of potential 


problems for further investigation. 


A. CONCLUSIONS 
1. Validation of the Short-Form UIS Instrument 
As mentioned, the most important conclusion is that 
the 13 question short-form instrument is inadequate for 
evaluation of overall UIS. Limitations associated with a 
fundamental change in the factor structure due to the 
Bigmttlcant impact of contractor services, unevenly 
distributed variables in a couple of factors, low internal 
consistency coefficient in one factor, no assessment of user 
interaction with the system, and inconsistent convergent 
validity findings all contributed to the inadequacy of this 
instrument for the evaluation of overall UIS of the CHCS. 
2. Areas of Satisfaction 
All three user groups (Physicians, Medical Support, 
and Administrative Support) were satisfied with the local 
Management Information Department (MID) staff and services, 
as well as the information product output. 
The Medical Support and the Administrative Support 
groups were satisfied with the training, understanding of the 


system, and level of participation. 
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3. Areas of Dissatisfaction 
All three user groups were dissatisfied with the 
contractor services to accomplish change to the system. 
Physicians were dissatisfied with the training, understanding 
of the system, and the level of participation. Additionally, 
Physicians were statistically less satisfied than the other 
two user groups for local MID staff and services, contractor 
services, and for the training, understanding of the systen, 
and level of participation. 
4. Time of Use Correlation 
The only significant correlation was the positive 
correlation exhibited by the Administrative Support group 
between the length of time of CHCS use and the information 
product output. There were no other significant correlations 


noted. 


B. FUTURE RESEARCH 
1. Re-design of the Short-form UIS Instrument 

Based on the findings from this study, a 12 question 
(Or variable) short-form UIS instrument, Appendix G, is 
suggested for future research investigation. In the 
demographics section, questions concerning education, age, 
gender, and previous computer use should be considered for 
elimination. Each of these user attributes have been found 
in a previous study [5], and for the most part supported by 
this study, not to correlate with UIS. In the questionnaire 
section, the use of an example to illustrate the scale 
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positions and their meanings would be highly beneficial for 
clarity. Baroudi and Orlikowski [2] offer a good example for 
use. A six interval Likert-type measurement scaling model 
vice the current seven interval model should be considered. 
Using a six interval model eliminates "neutral" responses 
without changing the scoring of the intervals (-3 to +3). 

Referring to Appendix G, each of the four factors 
would have three questions associated with it. Factor A (MID 
Staff and Services) would be unchanged and be represented 
with the same questions (in this case, questions 1, 6, and 
10). Factor B (Contractor Service) would consist of the same 
questions plus the added question about vendor support 
(questions 2, 8, and 11). Factor C (Information Input and 
Product) would consist of questions 7, 12, and the added 
question about "convenience of access" (question 9). Factor 
D (User Knowledge and Involvement) would be unchanged using 
questions 3, 4, and 5. Having equally distributed variables 
for each factor will prevent any one factor from biasing the 
overall score (which will now range from -36 to +36 vice -39 
to +39 due to one less question). 

Clarity of each question is important. Especially for 
the variable “convenience of access" where the emphasis is 
looking at the ease or difficulty with which the user may act 
to utilize the capability of the system. Bailey and Pearson 
[18] offer complete definitions for each of the questions. 


All of the current procedures described by Baroudi and 
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Orlikowski [2] for disseminating the instrument and ensuring 
confidentiality should be used. 
a. Validation of the New Short-form Instrument 

A large sampling of multiple hospitals (such as in 
this study) using the new short-form instrument will be 
necessary in order to validate the instrument. In addition 
to the methods used in this study (1.e., exploratory and 
confirmatory factor analysis, internal consistency testing, 
and equivalence testing), alternative forms of parallel tests 
(i.e., correlation with OT&E survey findings) or the 
split-half approach for estimates of reliability as described 
by Nunnally [27] should be used. Nunnally [27] does not 
recommend the retest method as an estimate of reliability as 
previous researchers have suggested. In addition, include 
sufficient time to allow a larger number of user interviews 
(minimum sample size of 20) for all three groups in order to 
conduct statistically meaningful convergent validity 


correlation study. 
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APPENDIX A 
Part A: General Information 


Hospital Department: (Check one) 
General Administration 
Nursing Administration 
Dietary 

Emergency 

Laboratory 

Outpatient Clinic 
Outpatient Nursing 
Pharmacy 

Radiology 

Other (Specify): 





LTT 





Job Description: (Check one) 
Clerk 

Corpsman (0000) 
Technician 

Nurse 

Pharmacist 

Physician 

Physician Assistant 
Other (Specify): 





TTT 


Highest Level of Education: (Check one) 
High School Graduate 

Some College 

Bachelor's Degree 

Some Graduate Work 

Master's Degree 

Doctoral Degree 

Medical Degree 

Other (Specify): 





iQ 
(D 


years 


Q 
(D 
= 
QO, 
(D 
Ky 


: Male Female 


Length of time (in months) you have used CHCS: 


Have you used other computer systems before ? 
If your answer was Yes, was it a health care 


information system ? Yes No 


ot 


© 


Part B: The Questionnaire 
This section of the survey conveys your own personal 
feelings concerning the use of the Composite Health Care 
system at Naval Hospital, (LOCATION). Please do not attempt 
to analyze the questions. Remember, there are no right or 
wrong answers. 
Please follow these instructions: 


a. Check each scale in the position that describes your 
evaluation of the factor being described. 


b. Check each scale, do not omit any. 

c. Check only one position for each scale. 

d. Check in the space, not between spaces. THIS, NOT THIS 
= ar wa x : 


e. Work rapidly. Rely on your first impressions. 


The scale positions are defined as follows: 


adjective X :  :  $:  $? :  : 073 0 adjecen es. 
(1) (2) (3) (4) (5) (6) (7) 
(1) extremely X (5) slightly Y 
(2) quite X (6) quite Y 
(3) slightly xX (7) extremely Y 
(4) meither X or Y; equally X or Y; does not apply 


ANSWERS BASED ON YOUR OWN FEELINGS 


1. Relationship with the Management Information Department 
(MID) staff 


dissonant : : =: _— 3: =: i_—S:__ |: «harmontene 
bad 3. 8 5 8 oes 
2 Processing of requests for changes to existing systems 
fast +: 3 8 aa ae 
untimely :. =: ..: 3 ee ee, 
3. Degree of training provided to users 
complete : 3 33 0): eee eee incomplete 
low : : : : : : : - hagh 


4. Your understanding of systems 


insufficient : : : : : : : * sufficient 


ecm a 


complete incomplete 


5. Your feeling of participation 


positive negative 


erence  e 


mSsuttLlelenc. : : : : ; : : : sufficient 


6. Attitude of the Management Information Department staff 


cooperative:  : =: =: =: =: =: ss belligerent 
Negative =. awe 8) GS positive 
7. Reliability of output information 
figs s wees: ss 3. low 
SUCWmOmlc es Ss 3 3  #$ : inferior 
Bee Relevancy of output information to intended function 


(degree of what user wants or requires and what is provided by 
the system) 


useful : : : : = : : : useless 


relevant : : : 2 : : . : irrelevant 


9. Accuracy of output information 


inaccurate : : : : : : ; > accurate 


low : : yee: : : : 5 Inlalejal 


Ams me eS GEES GSES SS 


10. Precision of output information (the variability of the 
output information from that which it purports to measure) 


low : : : : : : : 2 igaveflal 


mmm 


definite : : ; : ; : : > uncertain 


ee cm 


11. Communication with the Management Information Department 
staff 


dissonant : : ; : : : : s harmonious 


destructive : : : : : : : >: productive 


12. Time required for new systems development 


unreasonable :_ $$$ 3: “_W: #™ reasonapitc 
acceptable :  : : #$: #: : i: $: wunacceptabile 

13. Completeness of the output information 
sufficient :  : $: i: =: #3  %  : insutiterene 
adequate : _ :  _:  :  : =: =: =: inadequate 


Thank you for your cooperation 
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APPENDIX B 


Exploratory Factor Analysis SAS Commands: 


libname dataname "a:\"; 

title "CHCS UIS using Baroudi Short-Form Questionaire"; 
title2 "CHCS (3 hospitals combined) Factor Analysis"; 
option linesize=80; 

options pagesize=58; 


data 


proc 


DlEoc 


proc 


proc 


proc 


Proc 


proc 


dataname.dat (keep=Q1 Q2 Q3 04 Q5 Q6 Q7 O88 QY Q10 Q1l1 Q12 Q13); 
infile "A:\FA-CHCS.DAT"; 

input Q1 92 Q3 04 O5 Q6 Q7 O8 Q9 Q10 Q1l1 O12 Q13; 

factor method=ml 

msa /* Kaiser Measure of Sampling Adequacy - Sphericity */ 
scree 

heywood 

n=1 

rotate=varimax 

reorder; 

title3 "Maximum Likeliness Factor Analysis with One Factor"; 
factor method=ml 

heywood 

n=2 

rotate=varimax 

reorder; 

title3 "Maximum Likeliness Factor Analysis with Two Factors"; 
factor method=ml 

heywood 

n=3 

rotate=varimax 

reorder; 

title3 "Maximum Likeliness Factor Analysis with Three Factors"; 
factor method=ml 

heywood 

n=4 

rotate=varimax 

reorder; 

title3 "Maximum Likeliness Factor Analysis with Four Factors"; 
factor method=nl 

heywood 

n=5 

rotate=varimax 

reorder; 

title3 "Maximum Likeliness Factor Analysis with Five Factors"; 
standard mean=0 std=1 out=stdized; 

title3 "CHCS data set Standardized with Mean=0 and STD=1"; 


factor method=ml data=work.stdized 

heywood 

n=3 

rotate=varimax 

reorder; 

title3 "Max-Likeliness Factor Analysis w/3 Factors 


95 


(Standardized) "; 
proc factor method=ml data=work.stdized 
heywood 
n=4 
rotate=varimax 
reorder; 
title3 "Max-Likeliness Factor 
(Standardized)"; 
proc factor method=ml data=work.stdized 
heywood 
n=5 
rotate=varimax 
reorder; 
title3 "Max-Likeliness Factor 
(Standardized)"; 
run; 
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Analysis 


Analysis 


w/ 4 


w/5 


Factors 


Factors 


APPENDIX C 
Confirmatory Factor Analysis: 
PRELIS Commands: 


CHCS (3 hospitals combined) Factor Analysis Confirmation by 
LISREL 

DA NI=13 

LA 

QO1 O02 QO3 24 O5 Q6 Q7 OB QI O10 Q11 O12 O13 

RA FI=A:\FA-CHCS.DAT 

OR ALL 

OU MA=PM SM=A:\CHCS.PML SA=A:\CHCS.ACP PA 


LISREL Commands: 


FA of Ordinal Variables Confirmation by LISREL: 3 Hosps 
Combined 
DA NI=13 NO=340 MA=PM 
LA 
Q1 Q2 QO3 Q4 QO5 Q6 Q7 OB QY Q10 Q11 O12 O13 
PM FI=A:\CHCS. PML 
AC FI=A:\CHCS.ACP 
MO NX=13 NK=4 PH=ST 
LK 
FACTOR A FACTOR B FACTOR C FACTOR _D 
PR bX 1 1 LX 6 1 LX 11 1 continued 
LX 2 2 LX 12 2 continued 
iene s bx 8 3 bX 9 3 LX 10 3 LX 13 3 continued 
LX 3 4 LX 4 4 LX 5 4 
SOU SE TV 


| 


APPENDIX D 
Cronbach's Reliability Testing SAS Commands: 


libname dataname "a:\"; 
title "CHCS UIS usSing Baroudi Short-Form Questionaire"; 
title2 "CHCS (3 hospitals combined) Factor Analysis"; 
title3 "Cronbach alphas on factor clusters"; 
options linesize=80; 
options pagesize=58; 
data dataname.dat (keep=Q1 Q2 Q3 Q4 Q5 Q6 Q7 O8 OY Q10 Q11 Q12 
eps! 
infile "a:\FA-CHCS.dat"; 
input Q1 Q2 03 04 05 06 07 G8 OS O10 Clim OlZ cr 
proc corr alpha; 
var Q7 Q8 Q9 Q10 Q13; 
DEOC COre salpnay 
Var OFsOG Ol. 
DEOC COrE alpna: 
var 03 04 05; 
proc corr alpha; 
Var yO2 Ol. 
run; 
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APPENDIX E 


Equivalence Testing PRELIS/LISREL Commands: 


PRELIS Commands: 


PRELIS covariance matrix for Charleston Hospital 


DA NI=13 

LA 

ioe O03 04 O5 06 O77 O08 QY 
RA FI=A: \FA-CHAR. DAT 

OR ALL 


OU MA=CM SM=A:\CHAR-LIS.CMX 


OnCr OTe 2 O13 


PRELIS covariance matrix for Camp Lejeune Hospital 


DA NI=13 

ls 

Q1 Q2 Q3 24 Q5 Q6 Q7 QE8 QZ 
RA FI=A:\FA-CLEJ.DAT 

OR ALL 


OU MA=CM SM=A:\CLEJ-LIS.CMX 


OHO TOT O12 O13 


PRELIS covariance matrix for Jacksonville Hospital 


DA NI=13 
LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 O8 QI 
RA FI=A:\FA-JAX.DAT 

OR ALL 

OU MA=CM SM=A:\JAX-LIS.CMX 


LISREL Commands: 


HYPOTHESIS A, LOCATION: NH 
DA NG=3 NI=13 NO=101 

LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 QI 
CM FI=A:\CHAR-LIS.CMX 

MO NX=13 NK=13 LX=ID TD=ZE 
OU 

HYPOTHESIS A, LOCATION: NH 
DA NO=121 

LA 

Q1 Q2 Q3 24 Q5 Q6 Q7 O8 Q9 
CM FI=A:\CLEJ-LIS.CMX 

MO PH=IN 

OU 

HYPOTHESIS A, LOCATION: NH 
DA NO=118 

LA 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 O8 QI 
CM FI=A:\JAX-LIS.CMX 

MO PH=IN 


OF Olli ol2 O13 


CHARLESTON 


OLOSOTT O27 O13 


CAMP LEJEUNE 


OO O11 5O12 O13 


JACKSONVILLE 


OPO WOd EO 12 01.3 


oo 


OU 


HYPOTHESIS B, LOCATION: NH CHARLESTON 


DA 
LA 
OL 
CM 
MO 
lags 


NG=3 NI=13 NO=101 


O02 03 04 05 06° 0/3037 007 Oe O eros 


FI=A:\CHAR-LIS.CMX 
NX=13 NK=4 


LX 8 1 LX 9 1) EX=10 1 LX 135) contanued 


continued 


C 
3 

LX 3 4 LX 4 4 
7 1 KZ 2a 


HYPOTHESIS B, LOCATION: 


DA 
LA 
Op 
CM 
MO 
OU 


NO=121 


Q2 Q3 Q4 Q5 Q6 Q7 Q8 
FI=A: \CLEJ-LIS.CMX 
LX=PS 


HYPOTHESIS B, LOCATION: 


DA 
LA 
Q1 
CM 
MO 
OU 


NO=118 


O2 03 04°05 G6407 Oe 
FI=A: \JAX-LIS.CMX 
LX=Ps 


HYPOTHESIS C, LOCATION: 


DA 
LA 
Ql 
CM 


NG=3 NI=13 NO=101 


Q2 Q3 Q4 Q5 Q6 Q7 QB 
FI=A:\CHAR-LIS.CMX 


LX 6 3 continued 


il 


NH 


Q9 


NH 


Oo 


NH 


Q9 


ee 6) Gomer 


CAMP LEJEUNE 


O1T0vel) Cl25ere 


JACKSONVILLE 


O1ONOTIT Ol 2 201s 


CHARLESTON 


Omer Lire 2 01s 


LX 8 1 LX 9 1 LX 10 1 -LX 23) coneanucd 


MO NX=13 NK=4 
FR 
LX 2 2 continued 
LX 13 LX 6 3 continued 
LX 3 4 LX 4 4 
VA 1 LX 7 2 BX 12 2 LX Lies oes 
OU 
HYPOTHESIS C, NH CAMP LEJEUNE 
DA NO=121 
LA 
QO1 


CM 
MO 
OU 


O2 03 04°05 06 07 08 09 O10 Of CrZae1: 


FI=A:\CLEJ-LIS.CMX 
LX=IN 


HYPOTHESIS C, LOCATION: NH JACKSONVILLE 


DA 
LA 


NO=118 


Ql 02 03 04 05 06-07-03 "09 OCl0vell Claes 
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CM FI=A:\JAX-LIS.CMX 


MO 
OU 


LX=IN 


HYPOTHESIS D, LOCATION: NH CHARLESTON 
NG=3 NI=13 NoO=101 


DA 
LA 
Ql 
CM 
MO 
FR 


VA 
OU 


Q2 Q3 04 Q5 26 Q7 Q8 Q9 Q10 Q11 Q12 Q13 
FI=A:\CHAR-LIS.CMX 
NX=13 NK=4 


LX 8 1 
in 2 2 
me 1 3 
LX 3 4 
eax 7 


Ex 9 2 LX 10 1 EX 13 1 continued 
continued 

LX 6 3 continued 

LX 4 4 

1 LX 12 2 LX 11 3 LX 5 4 


HYPOTHESIS D, LOCATION: NH CAMP LEJEUNE 


DA 
LA 
Ql 
CM 
MO 
OU 


NO=121 


Q2 O03 24 Q5 O6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 
FI=A:\CLEJ-LIS.CMX 
LX=IN TD=IN 


HYPOTHESIS D, LOCATION: NH JACKSONVILLE 


DA 
LA 
Ql 
CM 
MO 
OU 


NO=118 


Genes 04 05 06 O07 O8 O09 O10 O11 O12 Q13 
FI=A:\JAX-LIS.CMX 
LX=IN TD=IN 


HYPOTHESIS E, LOCATION: NH CHARLESTON 
NG=3 NI=13 NO=101 


DA 
LA 


Ql 


fies O04 05 06 O7 OS O9 O10 O11 O12 O13 


CM FI=A: \CHAR-LIS.CMX 

MO NX=13 NK=4 : 

FR LX 8 1 LX 9 1 LX 10 1 LX 13 1 continued 
LX 2 2 continued 
LX 13 LX 6 3 continued 
LX 3 4 LX 4 4 

VA 1 LX 71 LX 12 2 LX 113 LX 5 4 

OU 

HYPOTHESIS E, LOCATION: NH CAMP LEJEUNE 

DA NO=121 

LA 

Q1 Q2 23 24 O5 26 O7 O8 OY O10 Q1l 012 O13 


CM 
MO 
OU 


FI=A: \CLEJ-LIS.CMX 
LX=IN TD=IN PH=IN 


HYPOTHESIS E, LOCATION: NH JACKSONVILLE 


DA 


NO=118 


101 


LA 

O1 Q2 03 04 05 06 07 Oc 095010 Cfo Zaor: 
CM FI=A:\JAX-LIS.CMX 

MO LX=IN TD=IN PH=IN 

OU 


BO 2 


APPENDIX F 


CHCS - COMBINED 3 HOSPITAL DATA 
SHORT-FORM UIS INSTRUMENT OVERALL STATISTICS 


TOTAL NUMBER OF SURVEYS: 340 
RESPONSE RATE (680 SENT) 50% 
MEDIAN EDUCATION LEVEL: a Some College 
AVERAGE AGE: 32 years 
MINIMUM AGE: 18 years 
MAXIMUM AGE: 61 years 
NUMBER OF MALES: 218 
NUMBER OF FEMALES: 127 
PERCENT MALES: 63% 
PERCENT FEMALES: 37% 
AVERAGE MONTHS USED: 2S 
MIN MONTHS USED: J. 
MAX MONTHS USED: 36 
NUMBER USED COMPUTER BEFORE 27 
PeReeNrT USED BEFORE: 80% 
NUMBER USED HLT CARE COMPTR 200 
PERCENT USED HC COMP BEFORE 59% 
PERCENT USED BEFORE AND 

oem HC SYSTEM BEFORE 74% 
NeeeeeERSN MONTHS USE 1-5: G7 20% 
Meee ERSN MONTHS USE 6-11: 98 29% 
Meme GRSN MONTHS USE 12-17: aidl 24% 
[emeeeERSN MONTHS USE 18 >: 94 28% 


OURS OURST 2 OUEST 3 QUEST 4 QUEST 5 OUEST 6 


AVERAGE ie =O. 41 Ore 5 0.72 0.68 1.43 
STD DEV 2 1.62 ie 3 Ibs (exe doe eS O 
MIN NUMBR = =3 =3 = 3 =3 =3 
MAX NUMBR 3 3 3 3 3 3 


DUES? OURS 6 “OURST 9 QUEST 10 QUEST 11 QUEST 12 


AVERAGE 1.00 O33 6 0.75 1.14 =On23 
STD DEV 1.42 1.41 1.38 ie 3 2 1.36 Wats 7 
MIN NUMBR =3 =3 = = = 3 =5 
MAX NUMBR 3 3 3 3 3 3 
QUEST 13 OVERALL 
AVERAGE OTe S226 
STD DEV i 3 1 OAS 1 
MIN NUMBR = 3 =29 
MAX NUMBR 3 39 
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MID CONTRACTOR INFORMATION KNOWLEDGE & 

STAFF/SERV STAFF/SERV PRODUCT INVOLVEMENT 

FACTOR A FACTOR B FACTOR C FACTOR D 

(1,6,11) (2,12) (7,8,9,10,13) (3,4,5) 
AVERAGE 1.28 055% 0.91 oon 
STD DEV tha dU, 1.43 aed 1gs35 
MIN NUMBR =e ~e = —3 
MAX NUMBR 3 3 3 3 
NAVAL HOSPITAL CHARLESTON, SC 

CHCS, SHORT-FORM UIS INSTRUMENT STATISTICS 


TOTAL NUMBER OF SURVEYS: 
RESPONSE RATE (180 SENT): 
MEDIAN EDUCATION LEVEL: 
AVERAGE AGE: 

MINIMUM AGE: 

MAXIMUM AGE: 

NUMBER OF MALES: (Percent) 
NUMBER OF FEMALES: 

AVERAGE MONTHS USED: 

MIN MONTHS USED: 

MAX MONTHS USED: 

NUMBER USED COMPUTER BEFORE 
PERCENT USED BEFORE: 

NUMBER USED HLT CARE COMPTR 
PERCENT USED HC COMP BEFORE 
PERCENT USED COMP BEFORE AN 
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56% 


Some college 


USED HC SYSTEM BEFORE: 
NO. PERSN MONTHS USE 1-5: 
NO. PERSN MONTHS USE 6-11: 
NO. PERSN MONTHS USE 12 >: 

QUEST 1 QUEST 22 

AVERAGE 1.68 -0.50 
STD DEV 22 67 
MIN NUMBR ee, =o 
MAX NUMBR 5 3 
GUEST. /_ FOUESI Ss 

AVERAGE 0379 0.96 
STD DEV 1.47 1.39 
MIN NUMBR =o 25 
MAX NUMBR 3 3 
QUEST 13 OVERALL 

AVERAGE 0.59 11.26 
STD DEV 1.44 Liss 
MIN NUMBR = =o 
MAX NUMBR 3 39 


=5 
3 
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QUEST 5 
P07 

IE 2a 

=3 

3 


QUEST 6 
Liao 
1.00 
-135 

3 


QUEST 10 QUEST 1] QUEST AE 


32 years 
19 years 
56 years 
57 (565) 
44 (44%) 
8.6 
ul 
19 
83 
82% 
aS 
58% 
D 
71% 
28 28% 
36 36% 
37 36 
QUEST 3) QUES Tea 
O57 ORo7 
deo 2 IL tele 
=3 =—5 
3 3 
QUEST 9 
ea 0.54 
i 10 1.43 


<3 
3 


Ne. 
1.14 
=a U) as. 

3 


-Ofze 
1.64 
=< 

3 


MID CONTRACTOR 
STAFF/SERV STAFF/SERV PRODUCT 
FACTOR A FACTOR B FACTOR C 
(i 621) (22 } (G7 soo). 13) 
AVERAGE 1.77 -0.36 0.81 
STD DEV 0.99 ae iL, iy; 
MIN NUMBR -1 -3 -3 
MAX NUMBR 3 3 3 


INFORMATION KNOWLEDGE & 


INVOLVEMENT 
FACTOR D 


NAVAL HOSPITAL CAMP LEJEUNE, NC 
GHesS)  SHeGRT-FORM US INSTRUMENT STATISTICS 


TOTAL NUMBER OF SURVEYS: 

RESPONSE RATE (250 SENT): 

MEDIAN EDUCATION LEVEL: 

AVERAGE AGE: 

MINIMUM AGE: 

MAXIMUM AGE: 

NUMBER OF MALES: (Percent) 

NUMBER OF FEMALES: (Percent) 

AVERAGE MONTHS USED: 

MIN MONTHS USED: 

MAX MONTHS USED: 

NUMBER USED COMPUTER BEFORE 

EeeeENT USED BEFORE: 

NUMBER USED HLT CARE COMPTR 

PERCENT USED HC COMP BEFORE 

PERCENT USED BEFORE AND 
Hot HC SYSTEM BEFORE 

NO. PERSN MONTHS USE 1-5: 

NO. PERSN MONTHS USE 6-11: 

NeeerERSN MONTHS USE 12-17: 

eee eERSN MONTHS USE 18 >: 


QUEST 1 QUEST’ 2 


AVERAGE 1.44 0.19 
STD DEV 15 ay) 1.41 
MIN NUMBR =e =3 
MAX NUMBR 3 3 
QUEST 7 QUEST 8 

AVERAGE eo 1.16 
STD DEV IR ene, Is 
MIN NUMBR =e} =3 
MAX NUMBR 3 3 


2 
48% 


Some college 


23° Years 

18 years 

61 years 

73 (60%) 

48 (40%) 
2.2 


QUEST 3 QUEST 4 QUEST 5 
0.44 Ozo7 0.84 
1.68 1.82 1.48 

= = =3 
se) 5 3 

QUESTSo OUEST 20>QOUEST 11 
IL gs She 0.86 Pe Sik 
L2G fine Ms Be, 

ag nape. = 3 


3 
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3 


3 


(3,4, 5) 


Oac/ 
Le Be 
=2 

3 


QUEST 46 
isos 
1.30 
— re. 

3 


QUBSL 12 
0.14 
no? 

= 3 
3 


QUEST 13 OVERALL 


AVERAGE 0.94 12.28 
STD DEV 1.46 12.41 
MIN NUMBR -3 -25 
MAX NUMBR 3 39 
MID CONTRACTOR INFORMATION KNOWLEDGE & 
STAFF/SERV STAFF/SERV PRODUCT INVOLVEMENT 
FACTOR A FACTOR B FACTOR C FACTOR D 
(lies6e ly) (212) (73, 943) (3,4,5) 
AVERAGE 1.46 Oey ele 0.65 
STD DEV ies iL) ie ik (017, Tes 
MIN NUMBR -1 -3 -3 -3 
MAX NUMBR 3 3 3 3 


NAVAL HOSPITAL JACKSONVILLE, FL 
CHCS, SHORT-FORM UIS INSTRUMENT STATISTICS 


TOTAL NUMBER OF SURVEYS: 118 
RESPONSE RATE (250 SENT): 47% 
MEDIAN EDUCATION LEVEL: BACHELOR DEGREE 
AVERAGE AGE: 32 YEARS 
MINIMUM AGE: 19 YEARS 
MAXIMUM AGE: 56 YEARS 
NUMBER OF MALES: (Percent) 83 (703) 
NUMBER OF FEMALES: (Percent) 35 (303) 
AVERAGE MONTHS USED: Sr 3 
MIN MONTHS USED: 1 
MAX MONTHS USED: Sz 
NUMBER USED COMPUTER BEFORE 102 
PERCENT USED BEFORE: 86% 
NUMBER USED HLT CARE COMPTR 78 
PERCENT USED HC COMP BEFORE 66% 
PERCENT USED BEFORE AND 

USED HC SYSTEM BEFORE 76% 
NO. PERSN MONTHS USE 1-5: 18 15% 
NO. PERSN MONTHS USE 6-11: 18 15% 
NO. PERSN MONTHS USE 12-17: 22 19% 
NO. PERSN MONTHS USE 18 >: 60 51% 


QUEST 1 QUEST 2 QUEST 3 QUEST 4 QUEST 5 QUECIE 


AVERAGE 0.74 -0.94 = Oho s 0.56 OL 17 0.74 
STD DEV 1 ae ne 1.60 1.58 1.55 1.23 
MIN NUMBR =3 =3 = = = 3 = 
MAX NUMBR 3 3 S| 3 3 3 
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QUEST 7 QUEST 8 QUEST 9 QUEST 10 QUEST 11 QUEST 
AVERAGE 0.89 0.67 0.94 0.81 0.52 -0. 
STD DEV 1.47 1.47 1.36 M22 1.25 ee 
MIN NUMBR -3 -3 -3 -3 = 
MAX NUMBR 3 3 3 3 3 
QUEST 13 OVERALL 
AVERAGE 0.52 4.46 
STD DEV 1.66 12.63 
MIN NUMBR -3 -28 
MAX NUMBR 3 36 
MID CONTRACTOR INFORMATION KNOWLEDGE & 
STAFF/SERV STAFF/SERV PRODUCT INVOLVEMENT 
FACTOR A FACTOR B FACTOR C FACTOR D 
(eG. 11" 2) (7S Oo 21613) (3,4,5) 
AVERAGE 0.67 -0.79 On77 0.07 
STD DEV 1.13 iL AG, 1.19 TO 
MIN NUMBR -3 -3 -3 -3 
MAX NUMBR 3 3 3 3 
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rZ 


me 
=5 


APPENDIX G 
Part A: General Information 


Hospital Department: (Check one) 
General Administration 
Nursing Administration 
Dietary 

Emergency 

Laboratory 

Outpatient Clinic 
Outpatient Nursing 
Pharmacy 

Radiology 

Other (Specify): 


LATTE 


Job Description: (Check one) 
Clerk 

Corpsman (0000) 
Technician 

Nurse 

Pharmacist 

Physician 

Physician Assistant 
Other (Specify): 





mm es ee ets ec ce cc ce mc cs cm sm mm sm me mc ee we oe eee eee ee ee ee oe ee oe 
eee ee ee ee ee ee ee es ee ee ee ee es ee eee 


Part B: The Questionnaire 


This section of the survey conveyS your own personal feelings 


concerning the use of the Composite Health Care system at Naval 
Hospital, (LOCATION). Please do not attempt to analyze the questions. 
Remember, there are no right or wrong answers. 


Please follow these instructions: 


a. Check each scale in the position that describes your evaluation 
of the factor being described. 
b. Check each scale, do not omit any. 
c. Check only one position for each scale. THIS, NOT THIS 
d. Check in the space, not between spaces. eae X : 
e. Work rapidly. Rely on your first impressions. 
The scale positions are defined as follows: 
adjective X adjective Y 
(Py) 2) (3) 4) Se 
(1) extremely X (4) slightly 
(2) quite X (5) .quitegy 
(Se Sieve lives. (6) extremely Y 
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The following example illustrates the scale positions and their 
meanings: 


My vacation in the Bahamas was: 


restful : : : : : > a hectic 


healthy ; x2 ; : : : unhealthy 


According to the responses, the person's vacation was extremely hectic 
and quite healthy 


ANSWERS BASED ON YOUR OWN FEELINGS 
1. Relationship with the Management Information Department (MID) 
staff 


AcsSonaneswe ts es 3. + >; harmonious 
Padme strom. 3s.  § good 
2. Processing of requests for changes to existing system 
Las: a) i - ees ns SLOW 
(mre ale Ss ae timely 
3. Degree of training provided to you 
SONplete (emt mo aweeee ss | ff incomplete 
LOVE tc ss  s high 
4. Your understanding of the system 
MieMinterent 3 +  : $$:  : #: =: sufficient 
complete incomplete 


5. Your feeling of participation 


positive : : : : : : >: negative 


insufficient : : : : : : wSuetrc lent 


6. Attitude of the Management Information Department staff 


cooperative : : : : : : : belligerent 


negative positive 


O09 


7. Reliability of output information (that Che output 15 relia 
timely, accurate, and precise) 


haggis : ; : : - : low 


Superior : ; ; : : : : inferior 


8. Vendor Support (the type and quality of the service rendered by 
the vendor) 


sufficient : : , ; : ; : >: insufficient 
poor : : : : : : : goo 
9. Convenience of access (the ease or difficulty that you have to 


utilize the capability of the system) 


easy ss $s 8 § #38 (see: dittiente 
efficient :  : : 3 3: + #&$. #3 “% anersieien. 
10. Communication with the Management Information Department staff 
dissonant :_ =: 3 3. ~— 3. 3 eehaenseue 
destructive : =: =: 3.3: 3 9: =: preoctcer. 
11. Time required for new system development 
unreasonable : :  : #$$} 3: + #: i #$: i : reasonable 
acceptable 3. 3) Se ee eee unacceptable 
12. Completeness of the output information 
sufficient =: 3: : $<: =: $3) §: 982 insult ves 
adequate : : =:  :  : =: =: =: inadequate 


Thanks again for your cooperation 
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